2026/04/20 08:26:20 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota
2026-04-20 08:26:20.201842 I | rookcmd: starting Rook v1.14.5 with arguments '/usr/local/bin/rook ceph operator'
2026-04-20 08:26:20.201879 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO
2026-04-20 08:26:20.201884 I | cephcmd: starting Rook-Ceph operator
2026-04-20 08:26:20.330508 I | cephcmd: base ceph version inside the rook operator image is "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)"
2026-04-20 08:26:20.339303 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var)
2026-04-20 08:26:20.339338 I | operator: watching all namespaces for Ceph CRs
2026-04-20 08:26:20.339447 I | operator: setting up schemes
2026-04-20 08:26:20.343156 I | operator: setting up the controller-runtime manager
2026-04-20 08:26:20.343674 I | ceph-cluster-controller: successfully started
2026-04-20 08:26:20.346633 I | op-k8sutil: ROOK_DISABLE_DEVICE_HOTPLUG="false" (env var)
2026-04-20 08:26:20.346648 I | ceph-cluster-controller: enabling hotplug orchestration
2026-04-20 08:26:20.346665 I | ceph-nodedaemon-controller: successfully started
2026-04-20 08:26:20.346681 I | ceph-block-pool-controller: successfully started
2026-04-20 08:26:20.346696 I | ceph-object-store-user-controller: successfully started
2026-04-20 08:26:20.346711 I | ceph-object-realm-controller: successfully started
2026-04-20 08:26:20.346722 I | ceph-object-zonegroup-controller: successfully started
2026-04-20 08:26:20.346745 I | ceph-object-zone-controller: successfully started
2026-04-20 08:26:20.346834 I | ceph-object-controller: successfully started
2026-04-20 08:26:20.346864 I | ceph-file-controller: successfully started
2026-04-20 08:26:20.346889 I | ceph-nfs-controller: successfully started
2026-04-20 08:26:20.346937 I | ceph-rbd-mirror-controller: successfully started
2026-04-20 08:26:20.346957 I | ceph-client-controller: successfully started
2026-04-20 08:26:20.346972 I | ceph-filesystem-mirror-controller: successfully started
2026-04-20 08:26:20.346994 I | operator: rook-ceph-operator-config-controller successfully started
2026-04-20 08:26:20.347009 I | ceph-csi: rook-ceph-operator-csi-controller successfully started
2026-04-20 08:26:20.347088 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started
2026-04-20 08:26:20.347107 I | ceph-bucket-topic: successfully started
2026-04-20 08:26:20.347116 I | ceph-bucket-notification: successfully started
2026-04-20 08:26:20.347127 I | ceph-bucket-notification: successfully started
2026-04-20 08:26:20.347136 I | ceph-fs-subvolumegroup-controller: successfully started
2026-04-20 08:26:20.347147 I | blockpool-rados-namespace-controller: successfully started
2026-04-20 08:26:20.347161 I | ceph-cosi-controller: successfully started
2026-04-20 08:26:20.347177 I | operator: starting the controller-runtime manager
2026-04-20 08:26:20.489998 I | op-k8sutil: ROOK_WATCH_FOR_NODE_FAILURE="true" (default)
2026-04-20 08:26:20.559820 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap)
2026-04-20 08:26:20.559848 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap)
2026-04-20 08:26:20.559870 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap)
2026-04-20 08:26:20.565408 I | op-k8sutil: ROOK_CEPH_ALLOW_LOOP_DEVICES="false" (configmap)
2026-04-20 08:26:20.565422 I | operator: rook-ceph-operator-config-controller done reconciling
2026-04-20 08:26:20.571948 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config"
2026-04-20 08:26:20.571993 I | op-k8sutil: ROOK_CSI_DISABLE_DRIVER="false" (configmap)
2026-04-20 08:26:20.572589 I | ceph-csi: CSI Ceph RBD driver disabled
2026-04-20 08:26:20.572606 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists
2026-04-20 08:26:20.574231 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists
2026-04-20 08:26:20.581244 I | ceph-csi: successfully removed CSI Ceph RBD driver
2026-04-20 08:26:20.581261 I | ceph-csi: CSI CephFS driver disabled
2026-04-20 08:26:20.581266 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists
2026-04-20 08:26:20.582998 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists
2026-04-20 08:26:20.595154 I | ceph-csi: successfully removed CSI CephFS driver
2026-04-20 08:26:20.595175 I | ceph-csi: CSI NFS driver disabled
2026-04-20 08:26:20.595182 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2026-04-20 08:26:20.597220 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2026-04-20 08:26:20.604218 I | ceph-csi: successfully removed CSI NFS driver
2026-04-20 08:26:42.319994 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "ceph"
2026-04-20 08:26:42.325998 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "ceph"
2026-04-20 08:26:42.327027 I | clusterdisruption-controller: deleted all legacy node drain canary pods
2026-04-20 08:26:42.334569 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 08:26:42.334592 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 08:26:42.334718 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap)
2026-04-20 08:26:42.334769 I | op-k8sutil: ROOK_OBC_PROVISIONER_NAME_PREFIX="" (default)
2026-04-20 08:26:42.334782 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "openstack.ceph.rook.io/bucket"
2026-04-20 08:26:42.335230 I | op-bucket-prov: successfully reconciled bucket provisioner
I0420 08:26:42.335347       1 manager.go:135] "msg"="starting provisioner" "logger"="objectbucket.io/provisioner-manager" "name"="openstack.ceph.rook.io/bucket"
2026-04-20 08:26:42.343426 I | ceph-cluster-controller: reconciling ceph cluster in namespace "openstack"
2026-04-20 08:26:42.365561 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (configmap)
2026-04-20 08:26:42.365591 I | op-k8sutil: CSI_DISABLE_HOLDER_PODS="true" (configmap)
2026-04-20 08:26:42.526316 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 08:26:42.526349 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 08:26:42.526376 I | ceph-spec: found the cluster info to connect to the external cluster. will use "client.admin" to check health and monitor status. mons=map[instance:0xc00190fb30]
2026-04-20 08:26:42.526396 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7...
2026-04-20 08:26:42.722857 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 08:26:42.722921 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 08:26:43.124042 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="false" (configmap)
2026-04-20 08:26:43.124084 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="false" (configmap)
2026-04-20 08:26:43.124093 I | op-k8sutil: ROOK_CSI_ENABLE_NFS="false" (configmap)
2026-04-20 08:26:43.124102 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (default)
2026-04-20 08:26:43.124110 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap)
2026-04-20 08:26:43.124118 I | op-k8sutil: CSI_GRPC_TIMEOUT_SECONDS="150" (configmap)
2026-04-20 08:26:43.124127 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default)
2026-04-20 08:26:43.124139 I | op-k8sutil: CSIADDONS_PORT="9070" (default)
2026-04-20 08:26:43.124147 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default)
2026-04-20 08:26:43.124154 I | op-k8sutil: CSI_ENABLE_LIVENESS="false" (default)
2026-04-20 08:26:43.124162 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="system-node-critical" (configmap)
2026-04-20 08:26:43.124171 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="system-cluster-critical" (configmap)
2026-04-20 08:26:43.124182 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (configmap)
2026-04-20 08:26:43.124191 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap)
2026-04-20 08:26:43.124199 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap)
2026-04-20 08:26:43.124206 I | op-k8sutil: CSI_ENABLE_NFS_SNAPSHOTTER="true" (configmap)
2026-04-20 08:26:43.126451 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap)
2026-04-20 08:26:43.126473 I | op-k8sutil: CSI_ENABLE_TOPOLOGY="false" (configmap)
2026-04-20 08:26:43.126481 I | op-k8sutil: CSI_ENABLE_ENCRYPTION="false" (configmap)
2026-04-20 08:26:43.126488 I | op-k8sutil: CSI_ENABLE_METADATA="false" (configmap)
2026-04-20 08:26:43.126497 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-20 08:26:43.126505 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default)
2026-04-20 08:26:43.126512 I | op-k8sutil: CSI_NFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-20 08:26:43.126520 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-20 08:26:43.126527 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default)
2026-04-20 08:26:43.126536 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap)
2026-04-20 08:26:43.126552 I | ceph-csi: Kubernetes version is 1.28
2026-04-20 08:26:43.126560 I | op-k8sutil: CSI_LOG_LEVEL="" (default)
2026-04-20 08:26:43.126567 I | op-k8sutil: CSI_SIDECAR_LOG_LEVEL="" (default)
2026-04-20 08:26:43.126573 I | op-k8sutil: CSI_LEADER_ELECTION_LEASE_DURATION="" (default)
2026-04-20 08:26:43.126580 I | op-k8sutil: CSI_LEADER_ELECTION_RENEW_DEADLINE="" (default)
2026-04-20 08:26:43.126588 I | op-k8sutil: CSI_LEADER_ELECTION_RETRY_PERIOD="" (default)
2026-04-20 08:26:43.530020 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.11.0" (configmap)
2026-04-20 08:26:43.530064 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1" (configmap)
2026-04-20 08:26:43.530269 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="registry.k8s.io/sig-storage/csi-provisioner:v4.0.1" (configmap)
2026-04-20 08:26:43.530282 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="registry.k8s.io/sig-storage/csi-attacher:v4.5.1" (configmap)
2026-04-20 08:26:43.530293 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="registry.k8s.io/sig-storage/csi-snapshotter:v7.0.2" (configmap)
2026-04-20 08:26:43.530341 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="registry.k8s.io/sig-storage/csi-resizer:v1.10.1" (configmap)
2026-04-20 08:26:43.530364 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default)
2026-04-20 08:26:43.530386 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.8.0" (configmap)
2026-04-20 08:26:43.530415 I | op-k8sutil: CSI_TOPOLOGY_DOMAIN_LABELS="" (default)
2026-04-20 08:26:43.530434 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default)
2026-04-20 08:26:43.530449 I | op-k8sutil: ROOK_CSI_NFS_POD_LABELS="" (default)
2026-04-20 08:26:43.530471 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default)
2026-04-20 08:26:43.530480 I | op-k8sutil: CSI_CLUSTER_NAME="" (default)
2026-04-20 08:26:43.530489 I | op-k8sutil: ROOK_CSI_IMAGE_PULL_POLICY="IfNotPresent" (configmap)
2026-04-20 08:26:43.530497 I | op-k8sutil: CSI_CEPHFS_KERNEL_MOUNT_OPTIONS="" (default)
2026-04-20 08:26:43.530511 I | op-k8sutil: CSI_CEPHFS_ATTACH_REQUIRED="true" (configmap)
2026-04-20 08:26:43.530520 I | op-k8sutil: CSI_RBD_ATTACH_REQUIRED="true" (configmap)
2026-04-20 08:26:43.530535 I | op-k8sutil: CSI_NFS_ATTACH_REQUIRED="true" (configmap)
2026-04-20 08:26:43.530557 I | op-k8sutil: CSI_DRIVER_NAME_PREFIX="rook-ceph" (default)
2026-04-20 08:26:43.536007 I | op-k8sutil: CSI_ENABLE_VOLUME_GROUP_SNAPSHOT="true" (configmap)
2026-04-20 08:26:43.536034 I | ceph-csi: skipping csi version check, since unsupported versions are allowed or csi is disabled
2026-04-20 08:26:43.536043 I | ceph-csi: CSI Ceph RBD driver disabled
2026-04-20 08:26:43.536060 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists
2026-04-20 08:26:43.539940 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists
2026-04-20 08:26:43.728825 I | ceph-csi: successfully removed CSI Ceph RBD driver
2026-04-20 08:26:43.728863 I | ceph-csi: CSI CephFS driver disabled
2026-04-20 08:26:43.728872 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists
2026-04-20 08:26:43.732260 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists
2026-04-20 08:26:43.939294 I | ceph-csi: successfully removed CSI CephFS driver
2026-04-20 08:26:43.939325 I | ceph-csi: CSI NFS driver disabled
2026-04-20 08:26:43.939334 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2026-04-20 08:26:43.941787 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2026-04-20 08:26:44.127408 I | ceph-csi: successfully removed CSI NFS driver
2026-04-20 08:27:00.124879 I | ceph-spec: detected ceph image version: "18.2.7-0 reef"
2026-04-20 08:27:00.124911 I | ceph-cluster-controller: validating ceph version from provided image
2026-04-20 08:27:00.130379 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 08:27:00.130396 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 08:27:00.133087 I | cephclient: writing config file /var/lib/rook/openstack/openstack.config
2026-04-20 08:27:00.133312 I | cephclient: generated admin config in /var/lib/rook/openstack
2026-04-20 08:27:00.788787 E | cephver: external cluster ceph version is a major version higher "18.2.7-0 reef" than the local cluster "0.0.0-0 <unknown version>", consider upgrading
2026-04-20 08:27:01.427105 I | ceph-cluster-controller: cluster "openstack": version "18.2.7-0 reef" detected for image "harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7"
2026-04-20 08:27:01.524770 I | ceph-cluster-controller: creating "rook-config-override" configmap
2026-04-20 08:27:01.527816 I | ceph-cluster-controller: creating "rook-ceph-config" secret
2026-04-20 08:27:01.540534 I | ceph-cluster-controller: external cluster identity established
2026-04-20 08:27:01.540562 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner"
2026-04-20 08:27:02.185093 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node"
2026-04-20 08:27:02.963375 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner"
2026-04-20 08:27:03.775160 I | clusterdisruption-controller: all PGs are active+clean. Restoring default OSD pdb settings
2026-04-20 08:27:03.775198 I | clusterdisruption-controller: creating the default pdb "rook-ceph-osd" with maxUnavailable=1 for all osd
2026-04-20 08:27:03.787402 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:27:04.193017 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node"
2026-04-20 08:27:05.073262 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:27:05.554198 I | ceph-csi: created kubernetes csi secrets for cluster "openstack"
2026-04-20 08:27:05.563919 I | ceph-cluster-controller: successfully updated csi config map
2026-04-20 08:27:05.563949 I | cephclient: getting or creating ceph auth key "client.crash"
2026-04-20 08:27:06.270996 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:27:06.572165 I | ceph-nodedaemon-controller: created kubernetes crash collector secret for cluster "openstack"
2026-04-20 08:27:06.575818 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "openstack"
2026-04-20 08:27:06.575839 I | ceph-cluster-controller: ceph status check interval is 1m0s
2026-04-20 08:27:06.575843 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "openstack"
2026-04-20 08:27:12.398455 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 08:27:12.398486 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 08:27:12.398503 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7...
2026-04-20 08:27:14.499557 I | ceph-spec: detected ceph image version: "18.2.7-0 reef"
2026-04-20 08:27:15.994149 I | ceph-object-controller: reconciling object store deployments
2026-04-20 08:27:16.012356 I | ceph-object-controller: ceph object store gateway service running at 10.105.42.233
2026-04-20 08:27:16.012393 I | ceph-object-controller: reconciling object store pools
2026-04-20 08:27:19.146870 I | cephclient: reconciling replicated pool ceph.rgw.control succeeded
2026-04-20 08:27:20.064600 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.control"
2026-04-20 08:27:20.064632 I | cephclient: updating pool "ceph.rgw.control" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.control_osd"
2026-04-20 08:27:20.064640 I | cephclient: crush rule "ceph.rgw.control" will no longer be used by pool "ceph.rgw.control"
2026-04-20 08:27:22.676066 I | cephclient: Successfully updated pool "ceph.rgw.control" failure domain to "osd"
2026-04-20 08:27:22.676094 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.control"
2026-04-20 08:27:28.215265 I | cephclient: reconciling replicated pool ceph.rgw.meta succeeded
2026-04-20 08:27:29.172952 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.meta"
2026-04-20 08:27:29.172996 I | cephclient: updating pool "ceph.rgw.meta" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.meta_osd"
2026-04-20 08:27:29.173005 I | cephclient: crush rule "ceph.rgw.meta" will no longer be used by pool "ceph.rgw.meta"
2026-04-20 08:27:31.264354 I | cephclient: Successfully updated pool "ceph.rgw.meta" failure domain to "osd"
2026-04-20 08:27:31.264395 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.meta"
2026-04-20 08:27:34.803874 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:27:35.991699 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:27:37.366373 I | cephclient: reconciling replicated pool ceph.rgw.log succeeded
2026-04-20 08:27:38.270779 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.log"
2026-04-20 08:27:38.270824 I | cephclient: updating pool "ceph.rgw.log" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.log_osd"
2026-04-20 08:27:38.270833 I | cephclient: crush rule "ceph.rgw.log" will no longer be used by pool "ceph.rgw.log"
2026-04-20 08:27:40.376635 I | cephclient: Successfully updated pool "ceph.rgw.log" failure domain to "osd"
2026-04-20 08:27:40.376665 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.log"
2026-04-20 08:27:44.425370 I | cephclient: reconciling replicated pool ceph.rgw.buckets.index succeeded
2026-04-20 08:27:45.331566 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.index"
2026-04-20 08:27:45.331619 I | cephclient: updating pool "ceph.rgw.buckets.index" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.index_osd"
2026-04-20 08:27:45.331630 I | cephclient: crush rule "ceph.rgw.buckets.index" will no longer be used by pool "ceph.rgw.buckets.index"
2026-04-20 08:27:47.481836 I | cephclient: Successfully updated pool "ceph.rgw.buckets.index" failure domain to "osd"
2026-04-20 08:27:47.481876 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.index"
2026-04-20 08:27:51.570163 I | cephclient: reconciling replicated pool ceph.rgw.buckets.non-ec succeeded
2026-04-20 08:27:53.479620 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.non-ec"
2026-04-20 08:27:53.479651 I | cephclient: updating pool "ceph.rgw.buckets.non-ec" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.non-ec_osd"
2026-04-20 08:27:53.479657 I | cephclient: crush rule "ceph.rgw.buckets.non-ec" will no longer be used by pool "ceph.rgw.buckets.non-ec"
2026-04-20 08:27:54.055738 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 08:27:55.580918 I | cephclient: Successfully updated pool "ceph.rgw.buckets.non-ec" failure domain to "osd"
2026-04-20 08:27:55.580960 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.non-ec"
2026-04-20 08:27:59.683569 I | cephclient: reconciling replicated pool ceph.rgw.otp succeeded
2026-04-20 08:28:00.601179 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.otp"
2026-04-20 08:28:00.601256 I | cephclient: updating pool "ceph.rgw.otp" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.otp_osd"
2026-04-20 08:28:00.601265 I | cephclient: crush rule "ceph.rgw.otp" will no longer be used by pool "ceph.rgw.otp"
2026-04-20 08:28:02.875765 I | cephclient: Successfully updated pool "ceph.rgw.otp" failure domain to "osd"
2026-04-20 08:28:02.875814 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.otp"
2026-04-20 08:28:05.851136 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:44} {StateName:creating+peering Count:3} {StateName:unknown Count:2}]"
2026-04-20 08:28:05.856859 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:28:06.983157 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:28:09.072863 I | cephclient: reconciling replicated pool .rgw.root succeeded
2026-04-20 08:28:10.099325 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule ".rgw.root"
2026-04-20 08:28:10.099358 I | cephclient: updating pool ".rgw.root" failure domain from "osd" to "osd" with new crush rule ".rgw.root_osd"
2026-04-20 08:28:10.099367 I | cephclient: crush rule ".rgw.root" will no longer be used by pool ".rgw.root"
2026-04-20 08:28:12.098016 I | cephclient: Successfully updated pool ".rgw.root" failure domain to "osd"
2026-04-20 08:28:12.098051 I | cephclient: setting pool property "pg_num_min" to "8" on pool ".rgw.root"
2026-04-20 08:28:16.165742 I | cephclient: reconciling replicated pool ceph.rgw.buckets.data succeeded
2026-04-20 08:28:17.068584 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.data"
2026-04-20 08:28:17.068629 I | cephclient: updating pool "ceph.rgw.buckets.data" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.data_osd"
2026-04-20 08:28:17.068635 I | cephclient: crush rule "ceph.rgw.buckets.data" will no longer be used by pool "ceph.rgw.buckets.data"
2026-04-20 08:28:19.221766 I | cephclient: Successfully updated pool "ceph.rgw.buckets.data" failure domain to "osd"
2026-04-20 08:28:19.221798 I | ceph-object-controller: configuring object store "ceph"
2026-04-20 08:28:19.892745 I | ceph-object-controller: Object store "ceph": realm=ceph, zonegroup=ceph, zone=ceph
2026-04-20 08:28:20.075243 I | ceph-object-controller: committing changes to RGW configuration period for CephObjectStore "openstack/ceph"
2026-04-20 08:28:20.501904 I | ceph-object-controller: configuration for object-store ceph is complete
2026-04-20 08:28:20.501998 I | ceph-object-controller: creating object store "ceph" in namespace "openstack"
2026-04-20 08:28:20.506079 I | cephclient: getting or creating ceph auth key "client.rgw.ceph.a"
2026-04-20 08:28:21.098487 I | ceph-object-controller: setting rgw config flags
2026-04-20 08:28:21.098534 I | op-config: setting "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-20 08:28:21.558240 I | op-config: successfully set "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-20 08:28:21.558281 I | op-config: setting "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-20 08:28:22.021971 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-20 08:28:22.022026 I | op-config: setting "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-20 08:28:22.491480 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-20 08:28:22.491524 I | op-config: setting "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-20 08:28:22.975567 I | op-config: successfully set "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-20 08:28:22.975607 I | op-config: setting "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-20 08:28:23.459247 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-20 08:28:23.459293 I | op-config: setting "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-20 08:28:23.930703 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-20 08:28:23.931097 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" created
2026-04-20 08:28:23.975819 I | ceph-object-controller: enabling rgw dashboard
2026-04-20 08:28:25.999253 I | ceph-object-controller: setting the dashboard api secret key
2026-04-20 08:28:26.014630 I | ceph-object-controller: created object store "ceph" in namespace "openstack"
2026-04-20 08:28:26.609449 I | ceph-object-controller: done setting the dashboard api secret key
2026-04-20 08:28:36.451089 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:28:37.557308 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:28:40.259167 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 08:28:56.449626 E | ceph-object-controller: failed to reconcile CephObjectStore "openstack/ceph". failed to create object store deployments: failed to get COSI user "cosi": Get "http://rook-ceph-rgw-ceph.openstack.svc:80/admin/user?format=json&uid=cosi": dial tcp 10.105.42.233:80: i/o timeout
2026-04-20 08:28:56.472143 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-20 08:28:56.472175 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-20 08:28:56.472198 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7...
2026-04-20 08:28:57.049447 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:28:57.638090 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:28:58.634851 I | ceph-spec: detected ceph image version: "18.2.7-0 reef"
2026-04-20 08:29:00.152195 I | ceph-object-controller: reconciling object store deployments
2026-04-20 08:29:00.184681 I | ceph-object-controller: ceph object store gateway service running at 10.105.42.233
2026-04-20 08:29:00.184716 I | ceph-object-controller: reconciling object store pools
2026-04-20 08:29:01.521715 I | cephclient: application "rgw" is already set on pool "ceph.rgw.control"
2026-04-20 08:29:01.521750 I | cephclient: reconciling replicated pool ceph.rgw.control succeeded
2026-04-20 08:29:02.469868 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.control"
2026-04-20 08:29:04.755435 I | cephclient: application "rgw" is already set on pool "ceph.rgw.meta"
2026-04-20 08:29:04.755469 I | cephclient: reconciling replicated pool ceph.rgw.meta succeeded
2026-04-20 08:29:05.685886 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.meta"
2026-04-20 08:29:07.671111 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:82} {StateName:unknown Count:31}]"
2026-04-20 08:29:07.675232 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:29:08.886215 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:82} {StateName:unknown Count:31}]"
2026-04-20 08:29:08.891140 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:29:09.161239 I | cephclient: application "rgw" is already set on pool "ceph.rgw.log"
2026-04-20 08:29:09.161281 I | cephclient: reconciling replicated pool ceph.rgw.log succeeded
2026-04-20 08:29:11.166307 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.log"
2026-04-20 08:29:13.900683 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.index"
2026-04-20 08:29:13.900959 I | cephclient: reconciling replicated pool ceph.rgw.buckets.index succeeded
2026-04-20 08:29:14.817645 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.index"
2026-04-20 08:29:16.928491 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.non-ec"
2026-04-20 08:29:16.928621 I | cephclient: reconciling replicated pool ceph.rgw.buckets.non-ec succeeded
2026-04-20 08:29:17.853024 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.non-ec"
2026-04-20 08:29:20.033185 I | cephclient: application "rgw" is already set on pool "ceph.rgw.otp"
2026-04-20 08:29:20.033223 I | cephclient: reconciling replicated pool ceph.rgw.otp succeeded
2026-04-20 08:29:20.962447 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.otp"
2026-04-20 08:29:23.046352 I | cephclient: application "rgw" is already set on pool ".rgw.root"
2026-04-20 08:29:23.046397 I | cephclient: reconciling replicated pool .rgw.root succeeded
2026-04-20 08:29:23.962474 I | cephclient: setting pool property "pg_num_min" to "8" on pool ".rgw.root"
2026-04-20 08:29:26.967208 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.data"
2026-04-20 08:29:26.967246 I | cephclient: reconciling replicated pool ceph.rgw.buckets.data succeeded
2026-04-20 08:29:27.854559 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 08:29:28.414842 I | ceph-object-controller: configuring object store "ceph"
2026-04-20 08:29:28.677254 I | ceph-object-controller: Object store "ceph": realm=ceph, zonegroup=ceph, zone=ceph
2026-04-20 08:29:28.887668 I | ceph-object-controller: there are no changes to commit for RGW configuration period for CephObjectStore "openstack/ceph"
2026-04-20 08:29:28.887708 I | ceph-object-controller: configuration for object-store ceph is complete
2026-04-20 08:29:28.887774 I | ceph-object-controller: creating object store "ceph" in namespace "openstack"
2026-04-20 08:29:28.893632 I | cephclient: getting or creating ceph auth key "client.rgw.ceph.a"
2026-04-20 08:29:29.499492 I | ceph-object-controller: setting rgw config flags
2026-04-20 08:29:29.499516 I | op-config: setting "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-20 08:29:29.973145 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-20 08:29:29.973178 I | op-config: setting "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-20 08:29:30.437017 I | op-config: successfully set "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-20 08:29:30.437058 I | op-config: setting "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-20 08:29:30.897508 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-20 08:29:30.897540 I | op-config: setting "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-20 08:29:31.366122 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-20 08:29:31.366155 I | op-config: setting "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-20 08:29:31.888593 I | op-config: successfully set "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-20 08:29:31.888623 I | op-config: setting "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-20 08:29:32.345830 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-20 08:29:32.346354 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" created
2026-04-20 08:29:32.365891 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" already exists. updating if needed
2026-04-20 08:29:32.376968 I | op-k8sutil: deployment "rook-ceph-rgw-ceph-a" did not change, nothing to update
2026-04-20 08:29:32.381388 I | ceph-object-controller: config map "rook-ceph-rgw-ceph-mime-types" for object store "ceph" already exists, not overwriting
2026-04-20 08:29:32.389313 I | ceph-object-controller: enabling rgw dashboard
2026-04-20 08:29:33.632953 I | ceph-object-controller: created object store "ceph" in namespace "openstack"
2026-04-20 08:29:34.620927 I | ceph-object-controller: creating COSI user "cosi"
2026-04-20 08:29:34.719038 I | ceph-spec: created ceph *v1.Secret object "rook-ceph-object-user-ceph-cosi"
2026-04-20 08:29:35.316947 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:29:38.252359 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:105} {StateName:peering Count:2}]"
2026-04-20 08:29:38.257588 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:29:39.465155 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:105} {StateName:peering Count:2}]"
2026-04-20 08:29:39.471145 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:30:08.858973 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:107} {StateName:unknown Count:24} {StateName:peering Count:4}]"
2026-04-20 08:30:08.864045 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:30:10.046794 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:123} {StateName:unknown Count:9} {StateName:peering Count:3}]"
2026-04-20 08:30:10.051077 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:30:14.046173 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 08:30:39.422691 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:30:40.624711 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:31:00.254192 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 08:31:10.021206 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:31:11.204484 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:31:40.606720 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:31:41.818175 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:31:46.397534 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 08:32:11.211196 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:32:12.415223 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:32:32.534630 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 08:32:41.818281 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:32:43.015570 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:33:12.385182 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:33:13.584878 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:33:18.670668 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 08:33:42.963102 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:33:44.215210 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:34:04.869549 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 08:34:13.526935 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:34:14.833869 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:34:44.086559 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:34:45.401948 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:34:51.012987 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 08:35:14.667096 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:35:15.962150 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:35:37.197320 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 08:35:45.236165 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:35:46.522687 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:36:15.813197 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:36:17.100714 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:36:23.335359 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 08:36:46.397061 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:36:47.724938 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:37:09.500756 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 08:37:16.988902 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:37:18.288820 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:37:47.596489 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:37:48.875804 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-20 08:37:55.673622 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-20 08:38:18.181471 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0