2026/04/20 12:07:07 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota
2026-04-20 12:07:07.370845 I | rookcmd: starting Rook v1.14.2 with arguments '/usr/local/bin/rook ceph operator'
2026-04-20 12:07:07.370872 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO
2026-04-20 12:07:07.370876 I | cephcmd: starting Rook-Ceph operator
2026-04-20 12:07:07.498123 I | cephcmd: base ceph version inside the rook operator image is "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)"
2026-04-20 12:07:07.505777 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var)
2026-04-20 12:07:07.505895 I | operator: watching all namespaces for Ceph CRs
2026-04-20 12:07:07.506101 I | operator: setting up schemes
2026-04-20 12:07:07.510146 I | operator: setting up the controller-runtime manager
2026-04-20 12:07:07.510578 I | ceph-cluster-controller: successfully started
2026-04-20 12:07:07.513261 I | op-k8sutil: ROOK_DISABLE_DEVICE_HOTPLUG="false" (env var)
2026-04-20 12:07:07.513273 I | ceph-cluster-controller: enabling hotplug orchestration
2026-04-20 12:07:07.513289 I | ceph-nodedaemon-controller: successfully started
2026-04-20 12:07:07.513305 I | ceph-block-pool-controller: successfully started
2026-04-20 12:07:07.513321 I | ceph-object-store-user-controller: successfully started
2026-04-20 12:07:07.513340 I | ceph-object-realm-controller: successfully started
2026-04-20 12:07:07.513350 I | ceph-object-zonegroup-controller: successfully started
2026-04-20 12:07:07.513359 I | ceph-object-zone-controller: successfully started
2026-04-20 12:07:07.513437 I | ceph-object-controller: successfully started
2026-04-20 12:07:07.513466 I | ceph-file-controller: successfully started
2026-04-20 12:07:07.513554 I | ceph-nfs-controller: successfully started
2026-04-20 12:07:07.513590 I | ceph-rbd-mirror-controller: successfully started
2026-04-20 12:07:07.513623 I | ceph-client-controller: successfully started
2026-04-20 12:07:07.513640 I | ceph-filesystem-mirror-controller: successfully started
2026-04-20 12:07:07.513662 I | operator: rook-ceph-operator-config-controller successfully started
2026-04-20 12:07:07.513678 I | ceph-csi: rook-ceph-operator-csi-controller successfully started
2026-04-20 12:07:07.513748 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started
2026-04-20 12:07:07.513761 I | ceph-bucket-topic: successfully started
2026-04-20 12:07:07.513768 I | ceph-bucket-notification: successfully started
2026-04-20 12:07:07.513778 I | ceph-bucket-notification: successfully started
2026-04-20 12:07:07.513785 I | ceph-fs-subvolumegroup-controller: successfully started
2026-04-20 12:07:07.513796 I | blockpool-rados-namespace-controller: successfully started
2026-04-20 12:07:07.513806 I | ceph-cosi-controller: successfully started
2026-04-20 12:07:07.513822 I | operator: starting the controller-runtime manager
2026-04-20 12:07:07.660637 I | op-k8sutil: ROOK_WATCH_FOR_NODE_FAILURE="true" (default)
2026-04-20 12:07:07.925406 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap)
2026-04-20 12:07:07.925442 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap)
2026-04-20 12:07:07.925463 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap)
2026-04-20 12:07:07.929070 I | op-k8sutil: ROOK_CEPH_ALLOW_LOOP_DEVICES="false" (configmap)
2026-04-20 12:07:07.929096 I | operator: rook-ceph-operator-config-controller done reconciling
2026-04-20 12:07:07.957814 I | op-k8sutil: ROOK_CSI_DISABLE_DRIVER="false" (configmap)
2026-04-20 12:07:07.961371 I | ceph-csi: CSI Ceph RBD driver disabled
2026-04-20 12:07:07.961392 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists
2026-04-20 12:07:07.964063 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists
2026-04-20 12:07:07.969458 I | ceph-csi: successfully removed CSI Ceph RBD driver
2026-04-20 12:07:07.969476 I | ceph-csi: CSI CephFS driver disabled
2026-04-20 12:07:07.969482 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists
2026-04-20 12:07:07.971507 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists
2026-04-20 12:07:07.977341 I | ceph-csi: successfully removed CSI CephFS driver
2026-04-20 12:07:07.977355 I | ceph-csi: CSI NFS driver disabled
2026-04-20 12:07:07.977359 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2026-04-20 12:07:07.978968 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2026-04-20 12:07:07.984312 I | ceph-csi: successfully removed CSI NFS driver
2026-04-20 12:07:28.942346 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "ceph"
2026-04-20 12:07:28.946090 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "ceph"
2026-04-20 12:07:28.947370 I | clusterdisruption-controller: deleted all legacy node drain canary pods
2026-04-20 12:07:28.966048 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 12:07:28.966074 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 12:07:28.966156 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap)
2026-04-20 12:07:28.966168 I | op-k8sutil: ROOK_OBC_PROVISIONER_NAME_PREFIX="" (default)
2026-04-20 12:07:28.966175 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "openstack.ceph.rook.io/bucket"
2026-04-20 12:07:28.966513 I | op-bucket-prov: successfully reconciled bucket provisioner
I0420 12:07:28.966587       1 manager.go:135] "msg"="starting provisioner" "logger"="objectbucket.io/provisioner-manager" "name"="openstack.ceph.rook.io/bucket"
2026-04-20 12:07:28.968572 I | ceph-cluster-controller: reconciling ceph cluster in namespace "openstack"
2026-04-20 12:07:28.972097 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config"
2026-04-20 12:07:28.977540 E | ceph-spec: failed to update cluster condition to {Type:Connecting Status:True Reason:ClusterConnecting Message:Attempting to connect to an external Ceph cluster LastHeartbeatTime:2026-04-20 12:07:28.968619221 +0000 UTC m=+21.635758440 LastTransitionTime:2026-04-20 12:07:28.968619131 +0000 UTC m=+21.635758350}. failed to update object "openstack/ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "ceph": the object has been modified; please apply your changes to the latest version and try again
2026-04-20 12:07:28.981617 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 12:07:28.981632 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 12:07:28.981646 I | ceph-spec: found the cluster info to connect to the external cluster. will use "client.admin" to check health and monitor status. mons=map[instance:0xc001926ae0]
2026-04-20 12:07:28.981657 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.1...
2026-04-20 12:07:28.982992 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (configmap)
2026-04-20 12:07:28.983007 I | op-k8sutil: CSI_DISABLE_HOLDER_PODS="true" (configmap)
2026-04-20 12:07:29.545863 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 12:07:29.545897 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 12:07:29.747778 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="false" (configmap)
2026-04-20 12:07:29.747806 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="false" (configmap)
2026-04-20 12:07:29.747811 I | op-k8sutil: ROOK_CSI_ENABLE_NFS="false" (configmap)
2026-04-20 12:07:29.747817 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (default)
2026-04-20 12:07:29.747823 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap)
2026-04-20 12:07:29.747828 I | op-k8sutil: CSI_GRPC_TIMEOUT_SECONDS="150" (configmap)
2026-04-20 12:07:29.747833 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default)
2026-04-20 12:07:29.747838 I | op-k8sutil: CSIADDONS_PORT="9070" (default)
2026-04-20 12:07:29.747843 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default)
2026-04-20 12:07:29.747853 I | op-k8sutil: CSI_ENABLE_LIVENESS="false" (default)
2026-04-20 12:07:29.747858 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="system-node-critical" (configmap)
2026-04-20 12:07:29.747863 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="system-cluster-critical" (configmap)
2026-04-20 12:07:29.747870 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (configmap)
2026-04-20 12:07:29.747876 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap)
2026-04-20 12:07:29.747881 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap)
2026-04-20 12:07:29.747885 I | op-k8sutil: CSI_ENABLE_NFS_SNAPSHOTTER="true" (configmap)
2026-04-20 12:07:29.750495 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap)
2026-04-20 12:07:29.750510 I | op-k8sutil: CSI_ENABLE_TOPOLOGY="false" (configmap)
2026-04-20 12:07:29.750513 I | op-k8sutil: CSI_ENABLE_ENCRYPTION="false" (configmap)
2026-04-20 12:07:29.750516 I | op-k8sutil: CSI_ENABLE_METADATA="false" (configmap)
2026-04-20 12:07:29.750520 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-20 12:07:29.750523 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default)
2026-04-20 12:07:29.750526 I | op-k8sutil: CSI_NFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-20 12:07:29.750529 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-20 12:07:29.750533 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default)
2026-04-20 12:07:29.750536 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap)
2026-04-20 12:07:29.750538 I | ceph-csi: Kubernetes version is 1.28
2026-04-20 12:07:29.750541 I | op-k8sutil: CSI_LOG_LEVEL="" (default)
2026-04-20 12:07:29.750544 I | op-k8sutil: CSI_SIDECAR_LOG_LEVEL="" (default)
2026-04-20 12:07:29.750548 I | op-k8sutil: CSI_LEADER_ELECTION_LEASE_DURATION="" (default)
2026-04-20 12:07:29.750551 I | op-k8sutil: CSI_LEADER_ELECTION_RENEW_DEADLINE="" (default)
2026-04-20 12:07:29.750554 I | op-k8sutil: CSI_LEADER_ELECTION_RETRY_PERIOD="" (default)
2026-04-20 12:07:29.948969 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.11.0" (configmap)
2026-04-20 12:07:29.949004 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.0" (configmap)
2026-04-20 12:07:29.949013 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="registry.k8s.io/sig-storage/csi-provisioner:v4.0.0" (configmap)
2026-04-20 12:07:29.949022 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="registry.k8s.io/sig-storage/csi-attacher:v4.5.0" (configmap)
2026-04-20 12:07:29.949029 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="registry.k8s.io/sig-storage/csi-snapshotter:v7.0.1" (configmap)
2026-04-20 12:07:29.949045 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="registry.k8s.io/sig-storage/csi-resizer:v1.10.0" (configmap)
2026-04-20 12:07:29.949060 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default)
2026-04-20 12:07:29.949073 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.8.0" (configmap)
2026-04-20 12:07:29.949081 I | op-k8sutil: CSI_TOPOLOGY_DOMAIN_LABELS="" (default)
2026-04-20 12:07:29.949089 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default)
2026-04-20 12:07:29.949096 I | op-k8sutil: ROOK_CSI_NFS_POD_LABELS="" (default)
2026-04-20 12:07:29.949102 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default)
2026-04-20 12:07:29.949109 I | op-k8sutil: CSI_CLUSTER_NAME="" (default)
2026-04-20 12:07:29.949115 I | op-k8sutil: ROOK_CSI_IMAGE_PULL_POLICY="IfNotPresent" (configmap)
2026-04-20 12:07:29.949122 I | op-k8sutil: CSI_CEPHFS_KERNEL_MOUNT_OPTIONS="" (default)
2026-04-20 12:07:29.949128 I | op-k8sutil: CSI_CEPHFS_ATTACH_REQUIRED="true" (configmap)
2026-04-20 12:07:29.949135 I | op-k8sutil: CSI_RBD_ATTACH_REQUIRED="true" (configmap)
2026-04-20 12:07:29.949142 I | op-k8sutil: CSI_NFS_ATTACH_REQUIRED="true" (configmap)
2026-04-20 12:07:29.949149 I | op-k8sutil: CSI_DRIVER_NAME_PREFIX="rook-ceph" (default)
2026-04-20 12:07:29.951822 I | op-k8sutil: CSI_ENABLE_VOLUME_GROUP_SNAPSHOT="true" (configmap)
2026-04-20 12:07:29.951835 I | ceph-csi: skipping csi version check, since unsupported versions are allowed or csi is disabled
2026-04-20 12:07:29.951840 I | ceph-csi: CSI Ceph RBD driver disabled
2026-04-20 12:07:29.951845 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists
2026-04-20 12:07:29.971009 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists
2026-04-20 12:07:30.148231 I | ceph-csi: successfully removed CSI Ceph RBD driver
2026-04-20 12:07:30.148363 I | ceph-csi: CSI CephFS driver disabled
2026-04-20 12:07:30.148373 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists
2026-04-20 12:07:30.150454 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists
2026-04-20 12:07:30.350198 I | ceph-csi: successfully removed CSI CephFS driver
2026-04-20 12:07:30.350227 I | ceph-csi: CSI NFS driver disabled
2026-04-20 12:07:30.350235 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2026-04-20 12:07:30.352947 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2026-04-20 12:07:30.548546 I | ceph-csi: successfully removed CSI NFS driver
2026-04-20 12:07:45.670554 I | ceph-spec: detected ceph image version: "18.2.1-0 reef"
2026-04-20 12:07:45.670579 I | ceph-cluster-controller: validating ceph version from provided image
2026-04-20 12:07:45.675762 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 12:07:45.675778 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 12:07:45.679929 I | cephclient: writing config file /var/lib/rook/openstack/openstack.config
2026-04-20 12:07:45.680050 I | cephclient: generated admin config in /var/lib/rook/openstack
2026-04-20 12:07:46.250332 E | cephver: external cluster ceph version is a major version higher "18.2.7-0 reef" than the local cluster "0.0.0-0 <unknown version>", consider upgrading
2026-04-20 12:07:46.819290 W | ceph-cluster-controller: image spec version 18.2.1-0 reef is lower than the running cluster version 18.2.7-0 reef, downgrading is not supported
2026-04-20 12:07:47.403630 I | ceph-cluster-controller: upgrading ceph cluster to "18.2.1-0 reef"
2026-04-20 12:07:47.403683 I | ceph-cluster-controller: cluster "openstack": version "18.2.1-0 reef" detected for image "harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.1"
2026-04-20 12:07:47.540470 I | ceph-cluster-controller: creating "rook-config-override" configmap
2026-04-20 12:07:47.543472 I | ceph-cluster-controller: creating "rook-ceph-config" secret
2026-04-20 12:07:47.556674 I | ceph-cluster-controller: external cluster identity established
2026-04-20 12:07:47.556753 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner"
2026-04-20 12:07:48.186403 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node"
2026-04-20 12:07:48.753436 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner"
2026-04-20 12:07:49.381214 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node"
2026-04-20 12:07:50.469612 I | clusterdisruption-controller: all PGs are active+clean. Restoring default OSD pdb settings
2026-04-20 12:07:50.469643 I | clusterdisruption-controller: creating the default pdb "rook-ceph-osd" with maxUnavailable=1 for all osd
2026-04-20 12:07:50.480929 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:07:50.715552 I | ceph-csi: created kubernetes csi secrets for cluster "openstack"
2026-04-20 12:07:50.746943 I | ceph-cluster-controller: successfully updated csi config map
2026-04-20 12:07:50.747035 I | cephclient: getting or creating ceph auth key "client.crash"
2026-04-20 12:07:51.682865 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:07:51.853289 I | ceph-nodedaemon-controller: created kubernetes crash collector secret for cluster "openstack"
2026-04-20 12:07:51.856315 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "openstack"
2026-04-20 12:07:51.856347 I | ceph-cluster-controller: ceph status check interval is 1m0s
2026-04-20 12:07:51.856352 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "openstack"
2026-04-20 12:07:59.037234 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 12:07:59.037270 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 12:07:59.037286 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.1...
2026-04-20 12:08:01.207397 I | ceph-spec: detected ceph image version: "18.2.1-0 reef"
2026-04-20 12:08:02.680899 I | ceph-object-controller: reconciling object store deployments
2026-04-20 12:08:02.696046 I | ceph-object-controller: ceph object store gateway service running at 10.100.253.96
2026-04-20 12:08:02.696090 I | ceph-object-controller: reconciling object store pools
2026-04-20 12:08:05.430418 I | cephclient: reconciling replicated pool ceph.rgw.control succeeded
2026-04-20 12:08:06.371013 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.control"
2026-04-20 12:08:06.371047 I | cephclient: updating pool "ceph.rgw.control" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.control_osd"
2026-04-20 12:08:06.371055 I | cephclient: crush rule "ceph.rgw.control" will no longer be used by pool "ceph.rgw.control"
2026-04-20 12:08:08.475754 I | cephclient: Successfully updated pool "ceph.rgw.control" failure domain to "osd"
2026-04-20 12:08:08.475783 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.control"
2026-04-20 12:08:12.862335 I | cephclient: reconciling replicated pool ceph.rgw.meta succeeded
2026-04-20 12:08:13.762869 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.meta"
2026-04-20 12:08:13.762979 I | cephclient: updating pool "ceph.rgw.meta" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.meta_osd"
2026-04-20 12:08:13.763013 I | cephclient: crush rule "ceph.rgw.meta" will no longer be used by pool "ceph.rgw.meta"
2026-04-20 12:08:15.908640 I | cephclient: Successfully updated pool "ceph.rgw.meta" failure domain to "osd"
2026-04-20 12:08:15.908760 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.meta"
2026-04-20 12:08:19.959515 I | cephclient: reconciling replicated pool ceph.rgw.log succeeded
2026-04-20 12:08:21.254324 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.log"
2026-04-20 12:08:21.254361 I | cephclient: updating pool "ceph.rgw.log" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.log_osd"
2026-04-20 12:08:21.254368 I | cephclient: crush rule "ceph.rgw.log" will no longer be used by pool "ceph.rgw.log"
2026-04-20 12:08:21.655678 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:17} {StateName:unknown Count:8}]"
2026-04-20 12:08:21.660462 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:08:22.528005 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:18} {StateName:unknown Count:7}]"
2026-04-20 12:08:22.531878 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:08:24.033180 I | cephclient: Successfully updated pool "ceph.rgw.log" failure domain to "osd"
2026-04-20 12:08:24.033221 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.log"
2026-04-20 12:08:28.131134 I | cephclient: reconciling replicated pool ceph.rgw.buckets.index succeeded
2026-04-20 12:08:29.063175 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.index"
2026-04-20 12:08:29.063209 I | cephclient: updating pool "ceph.rgw.buckets.index" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.index_osd"
2026-04-20 12:08:29.063217 I | cephclient: crush rule "ceph.rgw.buckets.index" will no longer be used by pool "ceph.rgw.buckets.index"
2026-04-20 12:08:31.238925 I | cephclient: Successfully updated pool "ceph.rgw.buckets.index" failure domain to "osd"
2026-04-20 12:08:31.238951 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.index"
2026-04-20 12:08:35.303057 I | cephclient: reconciling replicated pool ceph.rgw.buckets.non-ec succeeded
2026-04-20 12:08:36.199610 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.non-ec"
2026-04-20 12:08:36.199645 I | cephclient: updating pool "ceph.rgw.buckets.non-ec" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.non-ec_osd"
2026-04-20 12:08:36.199652 I | cephclient: crush rule "ceph.rgw.buckets.non-ec" will no longer be used by pool "ceph.rgw.buckets.non-ec"
2026-04-20 12:08:38.362155 I | cephclient: Successfully updated pool "ceph.rgw.buckets.non-ec" failure domain to "osd"
2026-04-20 12:08:38.362233 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.non-ec"
2026-04-20 12:08:38.676457 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 12:08:42.418051 I | cephclient: reconciling replicated pool ceph.rgw.otp succeeded
2026-04-20 12:08:43.289130 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.otp"
2026-04-20 12:08:43.289161 I | cephclient: updating pool "ceph.rgw.otp" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.otp_osd"
2026-04-20 12:08:43.289166 I | cephclient: crush rule "ceph.rgw.otp" will no longer be used by pool "ceph.rgw.otp"
2026-04-20 12:08:45.482244 I | cephclient: Successfully updated pool "ceph.rgw.otp" failure domain to "osd"
2026-04-20 12:08:45.482277 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.otp"
2026-04-20 12:08:49.543365 I | cephclient: reconciling replicated pool .rgw.root succeeded
2026-04-20 12:08:50.459757 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule ".rgw.root"
2026-04-20 12:08:50.459793 I | cephclient: updating pool ".rgw.root" failure domain from "osd" to "osd" with new crush rule ".rgw.root_osd"
2026-04-20 12:08:50.459801 I | cephclient: crush rule ".rgw.root" will no longer be used by pool ".rgw.root"
2026-04-20 12:08:52.639062 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:51} {StateName:unknown Count:6}]"
2026-04-20 12:08:52.643027 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:08:52.915094 I | cephclient: Successfully updated pool ".rgw.root" failure domain to "osd"
2026-04-20 12:08:52.915128 I | cephclient: setting pool property "pg_num_min" to "8" on pool ".rgw.root"
2026-04-20 12:08:54.176516 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:08:57.972272 I | cephclient: reconciling replicated pool ceph.rgw.buckets.data succeeded
2026-04-20 12:08:58.894894 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.data"
2026-04-20 12:08:58.894925 I | cephclient: updating pool "ceph.rgw.buckets.data" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.data_osd"
2026-04-20 12:08:58.894932 I | cephclient: crush rule "ceph.rgw.buckets.data" will no longer be used by pool "ceph.rgw.buckets.data"
2026-04-20 12:09:01.024655 I | cephclient: Successfully updated pool "ceph.rgw.buckets.data" failure domain to "osd"
2026-04-20 12:09:01.024712 I | ceph-object-controller: configuring object store "ceph"
2026-04-20 12:09:01.609529 I | ceph-object-controller: Object store "ceph": realm=ceph, zonegroup=ceph, zone=ceph
2026-04-20 12:09:01.788503 I | ceph-object-controller: committing changes to RGW configuration period for CephObjectStore "openstack/ceph"
2026-04-20 12:09:02.165143 I | ceph-object-controller: configuration for object-store ceph is complete
2026-04-20 12:09:02.165181 I | ceph-object-controller: creating object store "ceph" in namespace "openstack"
2026-04-20 12:09:02.169903 I | cephclient: getting or creating ceph auth key "client.rgw.ceph.a"
2026-04-20 12:09:02.758129 I | ceph-object-controller: setting rgw config flags
2026-04-20 12:09:02.758165 I | op-config: setting "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-20 12:09:03.207155 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-20 12:09:03.207199 I | op-config: setting "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-20 12:09:03.659938 I | op-config: successfully set "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-20 12:09:03.659977 I | op-config: setting "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-20 12:09:04.115818 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-20 12:09:04.115852 I | op-config: setting "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-20 12:09:04.563269 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-20 12:09:04.563308 I | op-config: setting "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-20 12:09:05.019689 I | op-config: successfully set "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-20 12:09:05.019728 I | op-config: setting "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-20 12:09:05.489785 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-20 12:09:05.490220 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" created
2026-04-20 12:09:05.530396 I | ceph-object-controller: enabling rgw dashboard
2026-04-20 12:09:07.517719 I | ceph-object-controller: created object store "ceph" in namespace "openstack"
2026-04-20 12:09:07.518738 I | ceph-object-controller: setting the dashboard api secret key
2026-04-20 12:09:08.110608 I | ceph-object-controller: done setting the dashboard api secret key
2026-04-20 12:09:23.247089 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:09:25.393013 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:09:25.497725 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 12:09:37.963706 E | ceph-object-controller: failed to reconcile CephObjectStore "openstack/ceph". failed to create object store deployments: failed to get COSI user "cosi": Get "http://rook-ceph-rgw-ceph.openstack.svc:80/admin/user?format=json&uid=cosi": dial tcp 10.100.253.96:80: i/o timeout
2026-04-20 12:09:37.996925 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 12:09:37.996957 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 12:09:37.996971 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.1...
2026-04-20 12:09:38.541388 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:09:39.105903 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:09:40.150420 I | ceph-spec: detected ceph image version: "18.2.1-0 reef"
2026-04-20 12:09:41.599815 I | ceph-object-controller: reconciling object store deployments
2026-04-20 12:09:41.627293 I | ceph-object-controller: ceph object store gateway service running at 10.100.253.96
2026-04-20 12:09:41.627323 I | ceph-object-controller: reconciling object store pools
2026-04-20 12:09:43.028323 I | cephclient: application "rgw" is already set on pool "ceph.rgw.control"
2026-04-20 12:09:43.028356 I | cephclient: reconciling replicated pool ceph.rgw.control succeeded
2026-04-20 12:09:43.937730 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.control"
2026-04-20 12:09:46.000104 I | cephclient: application "rgw" is already set on pool "ceph.rgw.meta"
2026-04-20 12:09:46.000143 I | cephclient: reconciling replicated pool ceph.rgw.meta succeeded
2026-04-20 12:09:46.918960 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.meta"
2026-04-20 12:09:49.011399 I | cephclient: application "rgw" is already set on pool "ceph.rgw.log"
2026-04-20 12:09:49.011433 I | cephclient: reconciling replicated pool ceph.rgw.log succeeded
2026-04-20 12:09:49.918943 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.log"
2026-04-20 12:09:52.114487 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.index"
2026-04-20 12:09:52.114517 I | cephclient: reconciling replicated pool ceph.rgw.buckets.index succeeded
2026-04-20 12:09:52.974240 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.index"
2026-04-20 12:09:54.185041 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:09:55.565992 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.non-ec"
2026-04-20 12:09:55.566015 I | cephclient: reconciling replicated pool ceph.rgw.buckets.non-ec succeeded
2026-04-20 12:09:57.356420 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:09:58.158102 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.non-ec"
2026-04-20 12:10:00.123051 I | cephclient: application "rgw" is already set on pool "ceph.rgw.otp"
2026-04-20 12:10:00.123083 I | cephclient: reconciling replicated pool ceph.rgw.otp succeeded
2026-04-20 12:10:01.032385 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.otp"
2026-04-20 12:10:03.206550 I | cephclient: application "rgw" is already set on pool ".rgw.root"
2026-04-20 12:10:03.206582 I | cephclient: reconciling replicated pool .rgw.root succeeded
2026-04-20 12:10:04.099518 I | cephclient: setting pool property "pg_num_min" to "8" on pool ".rgw.root"
2026-04-20 12:10:06.213879 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.data"
2026-04-20 12:10:06.213911 I | cephclient: reconciling replicated pool ceph.rgw.buckets.data succeeded
2026-04-20 12:10:07.100341 I | ceph-object-controller: configuring object store "ceph"
2026-04-20 12:10:07.326095 I | ceph-object-controller: Object store "ceph": realm=ceph, zonegroup=ceph, zone=ceph
2026-04-20 12:10:07.526164 I | ceph-object-controller: there are no changes to commit for RGW configuration period for CephObjectStore "openstack/ceph"
2026-04-20 12:10:07.526202 I | ceph-object-controller: configuration for object-store ceph is complete
2026-04-20 12:10:07.526209 I | ceph-object-controller: creating object store "ceph" in namespace "openstack"
2026-04-20 12:10:07.531745 I | cephclient: getting or creating ceph auth key "client.rgw.ceph.a"
2026-04-20 12:10:08.117842 I | ceph-object-controller: setting rgw config flags
2026-04-20 12:10:08.117869 I | op-config: setting "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-20 12:10:08.559902 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-20 12:10:08.559939 I | op-config: setting "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-20 12:10:09.015345 I | op-config: successfully set "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-20 12:10:09.015378 I | op-config: setting "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-20 12:10:09.454707 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-20 12:10:09.454745 I | op-config: setting "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-20 12:10:09.883153 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-20 12:10:09.883193 I | op-config: setting "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-20 12:10:10.322813 I | op-config: successfully set "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-20 12:10:10.322850 I | op-config: setting "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-20 12:10:11.048809 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-20 12:10:11.049183 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" created
2026-04-20 12:10:11.064349 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" already exists. updating if needed
2026-04-20 12:10:11.074559 I | op-k8sutil: deployment "rook-ceph-rgw-ceph-a" did not change, nothing to update
2026-04-20 12:10:11.078052 I | ceph-object-controller: config map "rook-ceph-rgw-ceph-mime-types" for object store "ceph" already exists, not overwriting
2026-04-20 12:10:11.084031 I | ceph-object-controller: enabling rgw dashboard
2026-04-20 12:10:12.860634 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 12:10:13.201858 I | ceph-object-controller: created object store "ceph" in namespace "openstack"
2026-04-20 12:10:15.119596 I | ceph-object-controller: creating COSI user "cosi"
2026-04-20 12:10:15.209600 I | ceph-spec: created ceph *v1.Secret object "rook-ceph-object-user-ceph-cosi"
2026-04-20 12:10:15.820385 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:10:24.745858 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:89} {StateName:unknown Count:8}]"
2026-04-20 12:10:24.749964 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:10:27.899132 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:94} {StateName:unknown Count:3}]"
2026-04-20 12:10:27.903097 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:10:55.309525 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:10:59.646012 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:11:00.653875 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 12:11:25.862712 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:11:30.204592 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:11:46.785289 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 12:11:56.448782 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:12:00.760057 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:12:27.018659 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:12:31.324754 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:12:32.965243 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 12:12:57.574423 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:13:01.886241 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:13:19.096127 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 12:13:28.142418 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:13:32.451877 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:13:58.695185 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:14:03.033804 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:14:05.684516 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 12:14:29.255329 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:14:33.619676 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:14:51.825006 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 12:14:59.819699 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:15:04.196117 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:15:30.414404 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:15:34.784966 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:15:37.948893 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 12:16:00.975461 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:16:05.373815 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:16:24.135459 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 12:16:31.548580 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:16:35.986985 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:17:02.125932 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:17:06.598547 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:17:10.281320 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 12:17:32.695770 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:17:37.173855 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:17:56.425292 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 12:18:03.293441 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:18:07.745134 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:18:33.856804 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:18:38.345985 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 12:18:42.538754 W | op-mon: external cluster mon count is 1, consider adding new monitors.