2026/04/19 11:22:16 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota
2026-04-19 11:22:16.520412 I | rookcmd: starting Rook v1.14.5 with arguments '/usr/local/bin/rook ceph operator'
2026-04-19 11:22:16.520451 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO
2026-04-19 11:22:16.520457 I | cephcmd: starting Rook-Ceph operator
2026-04-19 11:22:16.652536 I | cephcmd: base ceph version inside the rook operator image is "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)"
2026-04-19 11:22:16.661256 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var)
2026-04-19 11:22:16.661279 I | operator: watching all namespaces for Ceph CRs
2026-04-19 11:22:16.661447 I | operator: setting up schemes
2026-04-19 11:22:16.664954 I | operator: setting up the controller-runtime manager
2026-04-19 11:22:16.665342 I | ceph-cluster-controller: successfully started
2026-04-19 11:22:16.669056 I | op-k8sutil: ROOK_DISABLE_DEVICE_HOTPLUG="false" (env var)
2026-04-19 11:22:16.669106 I | ceph-cluster-controller: enabling hotplug orchestration
2026-04-19 11:22:16.669140 I | ceph-nodedaemon-controller: successfully started
2026-04-19 11:22:16.669173 I | ceph-block-pool-controller: successfully started
2026-04-19 11:22:16.669207 I | ceph-object-store-user-controller: successfully started
2026-04-19 11:22:16.669248 I | ceph-object-realm-controller: successfully started
2026-04-19 11:22:16.669279 I | ceph-object-zonegroup-controller: successfully started
2026-04-19 11:22:16.669310 I | ceph-object-zone-controller: successfully started
2026-04-19 11:22:16.669410 I | ceph-object-controller: successfully started
2026-04-19 11:22:16.669477 I | ceph-file-controller: successfully started
2026-04-19 11:22:16.669521 I | ceph-nfs-controller: successfully started
2026-04-19 11:22:16.669564 I | ceph-rbd-mirror-controller: successfully started
2026-04-19 11:22:16.669607 I | ceph-client-controller: successfully started
2026-04-19 11:22:16.669648 I | ceph-filesystem-mirror-controller: successfully started
2026-04-19 11:22:16.669677 I | operator: rook-ceph-operator-config-controller successfully started
2026-04-19 11:22:16.669712 I | ceph-csi: rook-ceph-operator-csi-controller successfully started
2026-04-19 11:22:16.669846 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started
2026-04-19 11:22:16.669897 I | ceph-bucket-topic: successfully started
2026-04-19 11:22:16.669922 I | ceph-bucket-notification: successfully started
2026-04-19 11:22:16.669955 I | ceph-bucket-notification: successfully started
2026-04-19 11:22:16.669990 I | ceph-fs-subvolumegroup-controller: successfully started
2026-04-19 11:22:16.670017 I | blockpool-rados-namespace-controller: successfully started
2026-04-19 11:22:16.670044 I | ceph-cosi-controller: successfully started
2026-04-19 11:22:16.670076 I | operator: starting the controller-runtime manager
2026-04-19 11:22:16.829622 I | op-k8sutil: ROOK_WATCH_FOR_NODE_FAILURE="true" (default)
2026-04-19 11:22:16.892887 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap)
2026-04-19 11:22:16.892913 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap)
2026-04-19 11:22:16.892931 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap)
2026-04-19 11:22:16.896869 I | op-k8sutil: ROOK_CEPH_ALLOW_LOOP_DEVICES="false" (configmap)
2026-04-19 11:22:16.896890 I | operator: rook-ceph-operator-config-controller done reconciling
2026-04-19 11:22:16.905336 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config"
2026-04-19 11:22:16.905368 I | op-k8sutil: ROOK_CSI_DISABLE_DRIVER="false" (configmap)
2026-04-19 11:22:16.906827 I | ceph-csi: CSI Ceph RBD driver disabled
2026-04-19 11:22:16.906843 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists
2026-04-19 11:22:16.909059 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists
2026-04-19 11:22:16.916599 I | ceph-csi: successfully removed CSI Ceph RBD driver
2026-04-19 11:22:16.916619 I | ceph-csi: CSI CephFS driver disabled
2026-04-19 11:22:16.916626 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists
2026-04-19 11:22:16.918614 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists
2026-04-19 11:22:16.924304 I | ceph-csi: successfully removed CSI CephFS driver
2026-04-19 11:22:16.924327 I | ceph-csi: CSI NFS driver disabled
2026-04-19 11:22:16.924333 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2026-04-19 11:22:16.927261 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2026-04-19 11:22:16.933736 I | ceph-csi: successfully removed CSI NFS driver
2026-04-19 11:22:38.940367 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "ceph"
2026-04-19 11:22:38.944909 I | clusterdisruption-controller: deleted all legacy node drain canary pods
2026-04-19 11:22:38.946883 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "ceph"
2026-04-19 11:22:38.954000 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-19 11:22:38.954035 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-19 11:22:38.954155 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap)
2026-04-19 11:22:38.954171 I | op-k8sutil: ROOK_OBC_PROVISIONER_NAME_PREFIX="" (default)
2026-04-19 11:22:38.954177 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "openstack.ceph.rook.io/bucket"
2026-04-19 11:22:38.954587 I | op-bucket-prov: successfully reconciled bucket provisioner
I0419 11:22:38.954677       1 manager.go:135] "msg"="starting provisioner" "logger"="objectbucket.io/provisioner-manager" "name"="openstack.ceph.rook.io/bucket"
2026-04-19 11:22:38.960533 I | ceph-cluster-controller: reconciling ceph cluster in namespace "openstack"
2026-04-19 11:22:38.970777 E | ceph-spec: failed to update cluster condition to {Type:Connecting Status:True Reason:ClusterConnecting Message:Attempting to connect to an external Ceph cluster LastHeartbeatTime:2026-04-19 11:22:38.960609327 +0000 UTC m=+22.474428822 LastTransitionTime:2026-04-19 11:22:38.960609207 +0000 UTC m=+22.474428712}. failed to update object "openstack/ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "ceph": the object has been modified; please apply your changes to the latest version and try again
2026-04-19 11:22:38.975871 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-19 11:22:38.975901 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-19 11:22:38.975924 I | ceph-spec: found the cluster info to connect to the external cluster. will use "client.admin" to check health and monitor status. mons=map[instance:0xc001825620]
2026-04-19 11:22:38.975945 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7...
2026-04-19 11:22:39.144310 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (configmap)
2026-04-19 11:22:39.144339 I | op-k8sutil: CSI_DISABLE_HOLDER_PODS="true" (configmap)
2026-04-19 11:22:39.745032 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-19 11:22:39.745065 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-19 11:22:39.944225 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="false" (configmap)
2026-04-19 11:22:39.944259 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="false" (configmap)
2026-04-19 11:22:39.944267 I | op-k8sutil: ROOK_CSI_ENABLE_NFS="false" (configmap)
2026-04-19 11:22:39.944275 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (default)
2026-04-19 11:22:39.944285 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap)
2026-04-19 11:22:39.944295 I | op-k8sutil: CSI_GRPC_TIMEOUT_SECONDS="150" (configmap)
2026-04-19 11:22:39.944307 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default)
2026-04-19 11:22:39.944319 I | op-k8sutil: CSIADDONS_PORT="9070" (default)
2026-04-19 11:22:39.944331 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default)
2026-04-19 11:22:39.944344 I | op-k8sutil: CSI_ENABLE_LIVENESS="false" (default)
2026-04-19 11:22:39.944362 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="system-node-critical" (configmap)
2026-04-19 11:22:39.944373 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="system-cluster-critical" (configmap)
2026-04-19 11:22:39.944389 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (configmap)
2026-04-19 11:22:39.944409 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap)
2026-04-19 11:22:39.944422 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap)
2026-04-19 11:22:39.944435 I | op-k8sutil: CSI_ENABLE_NFS_SNAPSHOTTER="true" (configmap)
2026-04-19 11:22:39.947443 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap)
2026-04-19 11:22:39.947469 I | op-k8sutil: CSI_ENABLE_TOPOLOGY="false" (configmap)
2026-04-19 11:22:39.947476 I | op-k8sutil: CSI_ENABLE_ENCRYPTION="false" (configmap)
2026-04-19 11:22:39.947483 I | op-k8sutil: CSI_ENABLE_METADATA="false" (configmap)
2026-04-19 11:22:39.947491 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-19 11:22:39.947501 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default)
2026-04-19 11:22:39.947508 I | op-k8sutil: CSI_NFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-19 11:22:39.947516 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-19 11:22:39.947523 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default)
2026-04-19 11:22:39.947531 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap)
2026-04-19 11:22:39.947538 I | ceph-csi: Kubernetes version is 1.28
2026-04-19 11:22:39.947545 I | op-k8sutil: CSI_LOG_LEVEL="" (default)
2026-04-19 11:22:39.947553 I | op-k8sutil: CSI_SIDECAR_LOG_LEVEL="" (default)
2026-04-19 11:22:39.947560 I | op-k8sutil: CSI_LEADER_ELECTION_LEASE_DURATION="" (default)
2026-04-19 11:22:39.947575 I | op-k8sutil: CSI_LEADER_ELECTION_RENEW_DEADLINE="" (default)
2026-04-19 11:22:39.947582 I | op-k8sutil: CSI_LEADER_ELECTION_RETRY_PERIOD="" (default)
2026-04-19 11:22:40.148794 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.11.0" (configmap)
2026-04-19 11:22:40.148830 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1" (configmap)
2026-04-19 11:22:40.148840 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="registry.k8s.io/sig-storage/csi-provisioner:v4.0.1" (configmap)
2026-04-19 11:22:40.148849 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="registry.k8s.io/sig-storage/csi-attacher:v4.5.1" (configmap)
2026-04-19 11:22:40.148865 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="registry.k8s.io/sig-storage/csi-snapshotter:v7.0.2" (configmap)
2026-04-19 11:22:40.148881 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="registry.k8s.io/sig-storage/csi-resizer:v1.10.1" (configmap)
2026-04-19 11:22:40.148898 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default)
2026-04-19 11:22:40.148913 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.8.0" (configmap)
2026-04-19 11:22:40.148922 I | op-k8sutil: CSI_TOPOLOGY_DOMAIN_LABELS="" (default)
2026-04-19 11:22:40.148933 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default)
2026-04-19 11:22:40.148951 I | op-k8sutil: ROOK_CSI_NFS_POD_LABELS="" (default)
2026-04-19 11:22:40.148956 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default)
2026-04-19 11:22:40.148962 I | op-k8sutil: CSI_CLUSTER_NAME="" (default)
2026-04-19 11:22:40.148969 I | op-k8sutil: ROOK_CSI_IMAGE_PULL_POLICY="IfNotPresent" (configmap)
2026-04-19 11:22:40.148975 I | op-k8sutil: CSI_CEPHFS_KERNEL_MOUNT_OPTIONS="" (default)
2026-04-19 11:22:40.148988 I | op-k8sutil: CSI_CEPHFS_ATTACH_REQUIRED="true" (configmap)
2026-04-19 11:22:40.148993 I | op-k8sutil: CSI_RBD_ATTACH_REQUIRED="true" (configmap)
2026-04-19 11:22:40.149000 I | op-k8sutil: CSI_NFS_ATTACH_REQUIRED="true" (configmap)
2026-04-19 11:22:40.149006 I | op-k8sutil: CSI_DRIVER_NAME_PREFIX="rook-ceph" (default)
2026-04-19 11:22:40.154021 I | op-k8sutil: CSI_ENABLE_VOLUME_GROUP_SNAPSHOT="true" (configmap)
2026-04-19 11:22:40.154036 I | ceph-csi: skipping csi version check, since unsupported versions are allowed or csi is disabled
2026-04-19 11:22:40.154047 I | ceph-csi: CSI Ceph RBD driver disabled
2026-04-19 11:22:40.154053 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists
2026-04-19 11:22:40.161222 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists
2026-04-19 11:22:40.389311 I | ceph-csi: successfully removed CSI Ceph RBD driver
2026-04-19 11:22:40.389337 I | ceph-csi: CSI CephFS driver disabled
2026-04-19 11:22:40.389344 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists
2026-04-19 11:22:40.392552 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists
2026-04-19 11:22:40.547490 I | ceph-csi: successfully removed CSI CephFS driver
2026-04-19 11:22:40.547519 I | ceph-csi: CSI NFS driver disabled
2026-04-19 11:22:40.547528 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2026-04-19 11:22:40.550157 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2026-04-19 11:22:40.747278 I | ceph-csi: successfully removed CSI NFS driver
2026-04-19 11:22:58.136009 I | ceph-spec: detected ceph image version: "18.2.7-0 reef"
2026-04-19 11:22:58.136039 I | ceph-cluster-controller: validating ceph version from provided image
2026-04-19 11:22:58.703777 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-19 11:22:58.703812 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-19 11:22:58.752775 I | cephclient: writing config file /var/lib/rook/openstack/openstack.config
2026-04-19 11:22:58.754294 I | cephclient: generated admin config in /var/lib/rook/openstack
2026-04-19 11:22:59.498880 E | cephver: external cluster ceph version is a major version higher "18.2.7-0 reef" than the local cluster "0.0.0-0 <unknown version>", consider upgrading
2026-04-19 11:23:00.171145 I | clusterdisruption-controller: all PGs are active+clean. Restoring default OSD pdb settings
2026-04-19 11:23:00.171188 I | clusterdisruption-controller: creating the default pdb "rook-ceph-osd" with maxUnavailable=1 for all osd
2026-04-19 11:23:00.790944 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:23:00.874964 I | ceph-cluster-controller: cluster "openstack": version "18.2.7-0 reef" detected for image "harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7"
2026-04-19 11:23:00.986040 I | ceph-cluster-controller: creating "rook-config-override" configmap
2026-04-19 11:23:00.989400 I | ceph-cluster-controller: creating "rook-ceph-config" secret
2026-04-19 11:23:01.009385 I | ceph-cluster-controller: external cluster identity established
2026-04-19 11:23:01.009423 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner"
2026-04-19 11:23:02.468402 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:23:03.465909 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:23:03.563581 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node"
2026-04-19 11:23:04.135615 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner"
2026-04-19 11:23:04.725947 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node"
2026-04-19 11:23:05.325035 I | ceph-csi: created kubernetes csi secrets for cluster "openstack"
2026-04-19 11:23:05.332856 I | ceph-cluster-controller: successfully updated csi config map
2026-04-19 11:23:05.332883 I | cephclient: getting or creating ceph auth key "client.crash"
2026-04-19 11:23:05.937351 I | ceph-nodedaemon-controller: created kubernetes crash collector secret for cluster "openstack"
2026-04-19 11:23:05.940151 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "openstack"
2026-04-19 11:23:05.940191 I | ceph-cluster-controller: ceph status check interval is 1m0s
2026-04-19 11:23:05.940197 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "openstack"
2026-04-19 11:23:09.049027 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-19 11:23:09.049052 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-19 11:23:09.049064 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7...
2026-04-19 11:23:10.928042 I | ceph-spec: detected ceph image version: "18.2.7-0 reef"
2026-04-19 11:23:12.435513 I | ceph-object-controller: reconciling object store deployments
2026-04-19 11:23:12.453667 I | ceph-object-controller: ceph object store gateway service running at 10.111.58.37
2026-04-19 11:23:12.453704 I | ceph-object-controller: reconciling object store pools
2026-04-19 11:23:15.547612 I | cephclient: reconciling replicated pool ceph.rgw.control succeeded
2026-04-19 11:23:16.476601 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.control"
2026-04-19 11:23:16.476635 I | cephclient: updating pool "ceph.rgw.control" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.control_osd"
2026-04-19 11:23:16.476643 I | cephclient: crush rule "ceph.rgw.control" will no longer be used by pool "ceph.rgw.control"
2026-04-19 11:23:18.604549 I | cephclient: Successfully updated pool "ceph.rgw.control" failure domain to "osd"
2026-04-19 11:23:18.604586 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.control"
2026-04-19 11:23:22.917217 I | cephclient: reconciling replicated pool ceph.rgw.meta succeeded
2026-04-19 11:23:23.876495 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.meta"
2026-04-19 11:23:23.876626 I | cephclient: updating pool "ceph.rgw.meta" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.meta_osd"
2026-04-19 11:23:23.876744 I | cephclient: crush rule "ceph.rgw.meta" will no longer be used by pool "ceph.rgw.meta"
2026-04-19 11:23:25.949627 I | cephclient: Successfully updated pool "ceph.rgw.meta" failure domain to "osd"
2026-04-19 11:23:25.949912 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.meta"
2026-04-19 11:23:30.270916 I | cephclient: reconciling replicated pool ceph.rgw.log succeeded
2026-04-19 11:23:31.675328 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.log"
2026-04-19 11:23:31.675368 I | cephclient: updating pool "ceph.rgw.log" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.log_osd"
2026-04-19 11:23:31.675376 I | cephclient: crush rule "ceph.rgw.log" will no longer be used by pool "ceph.rgw.log"
2026-04-19 11:23:31.988205 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:17} {StateName:unknown Count:7} {StateName:creating+peering Count:1}]"
2026-04-19 11:23:31.993769 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:23:33.092166 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:17} {StateName:unknown Count:7} {StateName:creating+peering Count:1}]"
2026-04-19 11:23:33.096010 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:23:34.112596 I | cephclient: Successfully updated pool "ceph.rgw.log" failure domain to "osd"
2026-04-19 11:23:34.112635 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.log"
2026-04-19 11:23:38.247597 I | cephclient: reconciling replicated pool ceph.rgw.buckets.index succeeded
2026-04-19 11:23:39.194023 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.index"
2026-04-19 11:23:39.194060 I | cephclient: updating pool "ceph.rgw.buckets.index" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.index_osd"
2026-04-19 11:23:39.194068 I | cephclient: crush rule "ceph.rgw.buckets.index" will no longer be used by pool "ceph.rgw.buckets.index"
2026-04-19 11:23:41.280800 I | cephclient: Successfully updated pool "ceph.rgw.buckets.index" failure domain to "osd"
2026-04-19 11:23:41.280839 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.index"
2026-04-19 11:23:45.349065 I | cephclient: reconciling replicated pool ceph.rgw.buckets.non-ec succeeded
2026-04-19 11:23:46.322560 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.non-ec"
2026-04-19 11:23:46.322589 I | cephclient: updating pool "ceph.rgw.buckets.non-ec" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.non-ec_osd"
2026-04-19 11:23:46.322597 I | cephclient: crush rule "ceph.rgw.buckets.non-ec" will no longer be used by pool "ceph.rgw.buckets.non-ec"
2026-04-19 11:23:48.405977 I | cephclient: Successfully updated pool "ceph.rgw.buckets.non-ec" failure domain to "osd"
2026-04-19 11:23:48.406017 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.non-ec"
2026-04-19 11:23:52.604724 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-19 11:23:54.501077 I | cephclient: reconciling replicated pool ceph.rgw.otp succeeded
2026-04-19 11:23:55.440838 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.otp"
2026-04-19 11:23:55.440878 I | cephclient: updating pool "ceph.rgw.otp" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.otp_osd"
2026-04-19 11:23:55.440884 I | cephclient: crush rule "ceph.rgw.otp" will no longer be used by pool "ceph.rgw.otp"
2026-04-19 11:23:57.515372 I | cephclient: Successfully updated pool "ceph.rgw.otp" failure domain to "osd"
2026-04-19 11:23:57.515416 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.otp"
2026-04-19 11:24:01.588049 I | cephclient: reconciling replicated pool .rgw.root succeeded
2026-04-19 11:24:03.087690 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule ".rgw.root"
2026-04-19 11:24:03.087732 I | cephclient: updating pool ".rgw.root" failure domain from "osd" to "osd" with new crush rule ".rgw.root_osd"
2026-04-19 11:24:03.087740 I | cephclient: crush rule ".rgw.root" will no longer be used by pool ".rgw.root"
2026-04-19 11:24:03.185077 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:49} {StateName:unknown Count:8}]"
2026-04-19 11:24:03.190049 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:24:04.264405 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:49} {StateName:unknown Count:8}]"
2026-04-19 11:24:04.269881 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:24:05.624435 I | cephclient: Successfully updated pool ".rgw.root" failure domain to "osd"
2026-04-19 11:24:05.624472 I | cephclient: setting pool property "pg_num_min" to "8" on pool ".rgw.root"
2026-04-19 11:24:10.727144 I | cephclient: reconciling replicated pool ceph.rgw.buckets.data succeeded
2026-04-19 11:24:11.655629 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.data"
2026-04-19 11:24:11.655673 I | cephclient: updating pool "ceph.rgw.buckets.data" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.data_osd"
2026-04-19 11:24:11.655682 I | cephclient: crush rule "ceph.rgw.buckets.data" will no longer be used by pool "ceph.rgw.buckets.data"
2026-04-19 11:24:15.840482 I | cephclient: Successfully updated pool "ceph.rgw.buckets.data" failure domain to "osd"
2026-04-19 11:24:15.840520 I | ceph-object-controller: configuring object store "ceph"
2026-04-19 11:24:16.711035 I | ceph-object-controller: Object store "ceph": realm=ceph, zonegroup=ceph, zone=ceph
2026-04-19 11:24:17.946161 I | ceph-object-controller: committing changes to RGW configuration period for CephObjectStore "openstack/ceph"
2026-04-19 11:24:18.454056 I | ceph-object-controller: configuration for object-store ceph is complete
2026-04-19 11:24:18.454100 I | ceph-object-controller: creating object store "ceph" in namespace "openstack"
2026-04-19 11:24:18.461573 I | cephclient: getting or creating ceph auth key "client.rgw.ceph.a"
2026-04-19 11:24:19.102028 I | ceph-object-controller: setting rgw config flags
2026-04-19 11:24:19.102073 I | op-config: setting "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-19 11:24:19.601147 I | op-config: successfully set "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-19 11:24:19.601175 I | op-config: setting "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-19 11:24:20.073747 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-19 11:24:20.073791 I | op-config: setting "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-19 11:24:20.562260 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-19 11:24:20.562301 I | op-config: setting "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-19 11:24:21.026632 I | op-config: successfully set "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-19 11:24:21.026690 I | op-config: setting "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-19 11:24:21.497027 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-19 11:24:21.497069 I | op-config: setting "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-19 11:24:21.992524 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-19 11:24:21.992882 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" created
2026-04-19 11:24:22.039745 I | ceph-object-controller: enabling rgw dashboard
2026-04-19 11:24:24.323353 I | ceph-object-controller: created object store "ceph" in namespace "openstack"
2026-04-19 11:24:24.324136 I | ceph-object-controller: setting the dashboard api secret key
2026-04-19 11:24:24.960812 I | ceph-object-controller: done setting the dashboard api secret key
2026-04-19 11:24:33.764636 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:24:34.864732 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:24:38.740982 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-19 11:24:54.706228 E | ceph-object-controller: failed to reconcile CephObjectStore "openstack/ceph". failed to create object store deployments: failed to get COSI user "cosi": Get "http://rook-ceph-rgw-ceph.openstack.svc:80/admin/user?format=json&uid=cosi": dial tcp 10.111.58.37:80: i/o timeout
2026-04-19 11:24:54.729052 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-19 11:24:54.729172 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-19 11:24:54.729218 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7...
2026-04-19 11:24:55.270600 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:24:56.149776 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:24:57.059332 I | ceph-spec: detected ceph image version: "18.2.7-0 reef"
2026-04-19 11:24:58.643112 I | ceph-object-controller: reconciling object store deployments
2026-04-19 11:24:58.685859 I | ceph-object-controller: ceph object store gateway service running at 10.111.58.37
2026-04-19 11:24:58.685899 I | ceph-object-controller: reconciling object store pools
2026-04-19 11:25:00.102241 I | cephclient: application "rgw" is already set on pool "ceph.rgw.control"
2026-04-19 11:25:00.102279 I | cephclient: reconciling replicated pool ceph.rgw.control succeeded
2026-04-19 11:25:01.103809 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.control"
2026-04-19 11:25:04.100740 I | cephclient: application "rgw" is already set on pool "ceph.rgw.meta"
2026-04-19 11:25:04.100783 I | cephclient: reconciling replicated pool ceph.rgw.meta succeeded
2026-04-19 11:25:04.983129 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:25:06.257549 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:25:06.283793 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.meta"
2026-04-19 11:25:08.939349 I | cephclient: application "rgw" is already set on pool "ceph.rgw.log"
2026-04-19 11:25:08.939387 I | cephclient: reconciling replicated pool ceph.rgw.log succeeded
2026-04-19 11:25:10.866593 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.log"
2026-04-19 11:25:14.113057 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.index"
2026-04-19 11:25:14.113104 I | cephclient: reconciling replicated pool ceph.rgw.buckets.index succeeded
2026-04-19 11:25:15.053351 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.index"
2026-04-19 11:25:17.131682 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.non-ec"
2026-04-19 11:25:17.131725 I | cephclient: reconciling replicated pool ceph.rgw.buckets.non-ec succeeded
2026-04-19 11:25:18.054043 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.non-ec"
2026-04-19 11:25:20.123217 I | cephclient: application "rgw" is already set on pool "ceph.rgw.otp"
2026-04-19 11:25:20.123255 I | cephclient: reconciling replicated pool ceph.rgw.otp succeeded
2026-04-19 11:25:21.089965 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.otp"
2026-04-19 11:25:23.306579 I | cephclient: application "rgw" is already set on pool ".rgw.root"
2026-04-19 11:25:23.306641 I | cephclient: reconciling replicated pool .rgw.root succeeded
2026-04-19 11:25:24.865416 I | cephclient: setting pool property "pg_num_min" to "8" on pool ".rgw.root"
2026-04-19 11:25:26.195597 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-19 11:25:27.500901 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.data"
2026-04-19 11:25:27.500939 I | cephclient: reconciling replicated pool ceph.rgw.buckets.data succeeded
2026-04-19 11:25:28.395711 I | ceph-object-controller: configuring object store "ceph"
2026-04-19 11:25:28.657044 I | ceph-object-controller: Object store "ceph": realm=ceph, zonegroup=ceph, zone=ceph
2026-04-19 11:25:28.847371 I | ceph-object-controller: there are no changes to commit for RGW configuration period for CephObjectStore "openstack/ceph"
2026-04-19 11:25:28.847405 I | ceph-object-controller: configuration for object-store ceph is complete
2026-04-19 11:25:28.847416 I | ceph-object-controller: creating object store "ceph" in namespace "openstack"
2026-04-19 11:25:28.852819 I | cephclient: getting or creating ceph auth key "client.rgw.ceph.a"
2026-04-19 11:25:29.481740 I | ceph-object-controller: setting rgw config flags
2026-04-19 11:25:29.481784 I | op-config: setting "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-19 11:25:29.940888 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-19 11:25:29.940935 I | op-config: setting "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-19 11:25:30.410678 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-19 11:25:30.410724 I | op-config: setting "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-19 11:25:30.876706 I | op-config: successfully set "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-19 11:25:30.876940 I | op-config: setting "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-19 11:25:31.351249 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-19 11:25:31.351301 I | op-config: setting "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-19 11:25:31.814143 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-19 11:25:31.814194 I | op-config: setting "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-19 11:25:32.295478 I | op-config: successfully set "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-19 11:25:32.295989 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" created
2026-04-19 11:25:32.311995 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" already exists. updating if needed
2026-04-19 11:25:32.322996 I | op-k8sutil: deployment "rook-ceph-rgw-ceph-a" did not change, nothing to update
2026-04-19 11:25:32.326826 I | ceph-object-controller: config map "rook-ceph-rgw-ceph-mime-types" for object store "ceph" already exists, not overwriting
2026-04-19 11:25:32.332425 I | ceph-object-controller: enabling rgw dashboard
2026-04-19 11:25:33.544370 I | ceph-object-controller: created object store "ceph" in namespace "openstack"
2026-04-19 11:25:34.447103 I | ceph-object-controller: creating COSI user "cosi"
2026-04-19 11:25:34.545331 I | ceph-spec: created ceph *v1.Secret object "rook-ceph-object-user-ceph-cosi"
2026-04-19 11:25:35.155308 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:25:35.733134 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:25:36.861944 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:26:05.779024 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:26:07.440362 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:26:13.266509 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-19 11:26:36.339530 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:26:38.003661 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:26:59.422385 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-19 11:27:06.905417 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:27:08.587010 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:27:37.486286 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:27:39.168739 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:27:45.632722 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-19 11:28:08.040531 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:28:09.746144 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:28:31.774877 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-19 11:28:38.663432 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:28:40.317279 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:29:09.234247 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:29:10.908260 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:29:18.542788 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-19 11:29:39.826046 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:29:41.493747 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:30:04.732336 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-19 11:30:10.419001 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:30:12.043842 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:30:41.017349 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:30:42.632304 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:30:50.894809 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-19 11:31:11.578984 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:31:13.203393 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:31:37.071892 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-19 11:31:42.168353 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:31:43.830805 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:32:12.785858 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:32:14.400515 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:32:23.235380 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-19 11:32:43.361882 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:32:44.976362 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:33:09.388690 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-19 11:33:13.939040 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:33:15.546402 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:33:44.532050 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:33:46.098783 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:33:55.524867 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-19 11:34:15.116204 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-19 11:34:16.668293 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0