2026/04/11 18:02:55 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota
2026-04-11 18:02:55.533130 I | rookcmd: starting Rook v1.14.5 with arguments '/usr/local/bin/rook ceph operator'
2026-04-11 18:02:55.533160 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO
2026-04-11 18:02:55.533163 I | cephcmd: starting Rook-Ceph operator
2026-04-11 18:02:55.663768 I | cephcmd: base ceph version inside the rook operator image is "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)"
2026-04-11 18:02:55.672596 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var)
2026-04-11 18:02:55.672620 I | operator: watching all namespaces for Ceph CRs
2026-04-11 18:02:55.672841 I | operator: setting up schemes
2026-04-11 18:02:55.676304 I | operator: setting up the controller-runtime manager
2026-04-11 18:02:55.676965 I | ceph-cluster-controller: successfully started
2026-04-11 18:02:55.679969 I | op-k8sutil: ROOK_DISABLE_DEVICE_HOTPLUG="false" (env var)
2026-04-11 18:02:55.679983 I | ceph-cluster-controller: enabling hotplug orchestration
2026-04-11 18:02:55.679998 I | ceph-nodedaemon-controller: successfully started
2026-04-11 18:02:55.680010 I | ceph-block-pool-controller: successfully started
2026-04-11 18:02:55.680028 I | ceph-object-store-user-controller: successfully started
2026-04-11 18:02:55.680041 I | ceph-object-realm-controller: successfully started
2026-04-11 18:02:55.680049 I | ceph-object-zonegroup-controller: successfully started
2026-04-11 18:02:55.680062 I | ceph-object-zone-controller: successfully started
2026-04-11 18:02:55.680142 I | ceph-object-controller: successfully started
2026-04-11 18:02:55.680171 I | ceph-file-controller: successfully started
2026-04-11 18:02:55.680195 I | ceph-nfs-controller: successfully started
2026-04-11 18:02:55.680222 I | ceph-rbd-mirror-controller: successfully started
2026-04-11 18:02:55.680258 I | ceph-client-controller: successfully started
2026-04-11 18:02:55.680275 I | ceph-filesystem-mirror-controller: successfully started
2026-04-11 18:02:55.680291 I | operator: rook-ceph-operator-config-controller successfully started
2026-04-11 18:02:55.680301 I | ceph-csi: rook-ceph-operator-csi-controller successfully started
2026-04-11 18:02:55.680370 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started
2026-04-11 18:02:55.680386 I | ceph-bucket-topic: successfully started
2026-04-11 18:02:55.680392 I | ceph-bucket-notification: successfully started
2026-04-11 18:02:55.680403 I | ceph-bucket-notification: successfully started
2026-04-11 18:02:55.680411 I | ceph-fs-subvolumegroup-controller: successfully started
2026-04-11 18:02:55.680437 I | blockpool-rados-namespace-controller: successfully started
2026-04-11 18:02:55.680449 I | ceph-cosi-controller: successfully started
2026-04-11 18:02:55.680465 I | operator: starting the controller-runtime manager
2026-04-11 18:02:55.822673 I | op-k8sutil: ROOK_WATCH_FOR_NODE_FAILURE="true" (default)
2026-04-11 18:02:55.904743 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap)
2026-04-11 18:02:55.904785 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap)
2026-04-11 18:02:55.904805 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap)
2026-04-11 18:02:55.908010 I | op-k8sutil: ROOK_CEPH_ALLOW_LOOP_DEVICES="false" (configmap)
2026-04-11 18:02:55.908037 I | operator: rook-ceph-operator-config-controller done reconciling
2026-04-11 18:02:55.921145 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config"
2026-04-11 18:02:55.921204 I | op-k8sutil: ROOK_CSI_DISABLE_DRIVER="false" (configmap)
2026-04-11 18:02:55.922356 I | ceph-csi: CSI Ceph RBD driver disabled
2026-04-11 18:02:55.922375 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists
2026-04-11 18:02:55.925586 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists
2026-04-11 18:02:55.932205 I | ceph-csi: successfully removed CSI Ceph RBD driver
2026-04-11 18:02:55.932231 I | ceph-csi: CSI CephFS driver disabled
2026-04-11 18:02:55.932240 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists
2026-04-11 18:02:55.934669 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists
2026-04-11 18:02:55.941434 I | ceph-csi: successfully removed CSI CephFS driver
2026-04-11 18:02:55.941450 I | ceph-csi: CSI NFS driver disabled
2026-04-11 18:02:55.941456 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2026-04-11 18:02:55.943465 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2026-04-11 18:02:55.949154 I | ceph-csi: successfully removed CSI NFS driver
2026-04-11 18:03:17.716102 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "ceph"
2026-04-11 18:03:17.724168 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "ceph"
2026-04-11 18:03:17.725176 I | clusterdisruption-controller: deleted all legacy node drain canary pods
2026-04-11 18:03:17.733413 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-11 18:03:17.733440 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-11 18:03:17.733628 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap)
2026-04-11 18:03:17.733642 I | op-k8sutil: ROOK_OBC_PROVISIONER_NAME_PREFIX="" (default)
2026-04-11 18:03:17.733647 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "openstack.ceph.rook.io/bucket"
2026-04-11 18:03:17.734589 I | op-bucket-prov: successfully reconciled bucket provisioner
I0411 18:03:17.734751       1 manager.go:135] "msg"="starting provisioner" "logger"="objectbucket.io/provisioner-manager" "name"="openstack.ceph.rook.io/bucket"
2026-04-11 18:03:17.737726 I | ceph-cluster-controller: reconciling ceph cluster in namespace "openstack"
2026-04-11 18:03:17.760504 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-11 18:03:17.760533 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-11 18:03:17.760578 I | ceph-spec: found the cluster info to connect to the external cluster. will use "client.admin" to check health and monitor status. mons=map[instance:0xc0015c9c50]
2026-04-11 18:03:17.760603 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7...
2026-04-11 18:03:17.761695 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (configmap)
2026-04-11 18:03:17.761710 I | op-k8sutil: CSI_DISABLE_HOLDER_PODS="true" (configmap)
2026-04-11 18:03:18.523384 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-11 18:03:18.523422 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-11 18:03:18.723144 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="false" (configmap)
2026-04-11 18:03:18.723169 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="false" (configmap)
2026-04-11 18:03:18.723174 I | op-k8sutil: ROOK_CSI_ENABLE_NFS="false" (configmap)
2026-04-11 18:03:18.723181 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (default)
2026-04-11 18:03:18.723187 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap)
2026-04-11 18:03:18.723192 I | op-k8sutil: CSI_GRPC_TIMEOUT_SECONDS="150" (configmap)
2026-04-11 18:03:18.723197 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default)
2026-04-11 18:03:18.723202 I | op-k8sutil: CSIADDONS_PORT="9070" (default)
2026-04-11 18:03:18.723207 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default)
2026-04-11 18:03:18.723212 I | op-k8sutil: CSI_ENABLE_LIVENESS="false" (default)
2026-04-11 18:03:18.723217 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="system-node-critical" (configmap)
2026-04-11 18:03:18.723222 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="system-cluster-critical" (configmap)
2026-04-11 18:03:18.723231 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (configmap)
2026-04-11 18:03:18.723237 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap)
2026-04-11 18:03:18.723242 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="true" (configmap)
2026-04-11 18:03:18.723247 I | op-k8sutil: CSI_ENABLE_NFS_SNAPSHOTTER="true" (configmap)
2026-04-11 18:03:18.725997 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap)
2026-04-11 18:03:18.726012 I | op-k8sutil: CSI_ENABLE_TOPOLOGY="false" (configmap)
2026-04-11 18:03:18.726017 I | op-k8sutil: CSI_ENABLE_ENCRYPTION="false" (configmap)
2026-04-11 18:03:18.726022 I | op-k8sutil: CSI_ENABLE_METADATA="false" (configmap)
2026-04-11 18:03:18.726028 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-11 18:03:18.726033 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default)
2026-04-11 18:03:18.726038 I | op-k8sutil: CSI_NFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-11 18:03:18.726042 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2026-04-11 18:03:18.726048 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default)
2026-04-11 18:03:18.726053 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap)
2026-04-11 18:03:18.726057 I | ceph-csi: Kubernetes version is 1.28
2026-04-11 18:03:18.726061 I | op-k8sutil: CSI_LOG_LEVEL="" (default)
2026-04-11 18:03:18.726066 I | op-k8sutil: CSI_SIDECAR_LOG_LEVEL="" (default)
2026-04-11 18:03:18.726071 I | op-k8sutil: CSI_LEADER_ELECTION_LEASE_DURATION="" (default)
2026-04-11 18:03:18.726075 I | op-k8sutil: CSI_LEADER_ELECTION_RENEW_DEADLINE="" (default)
2026-04-11 18:03:18.726086 I | op-k8sutil: CSI_LEADER_ELECTION_RETRY_PERIOD="" (default)
2026-04-11 18:03:18.925375 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.11.0" (configmap)
2026-04-11 18:03:18.925472 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1" (configmap)
2026-04-11 18:03:18.925516 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="registry.k8s.io/sig-storage/csi-provisioner:v4.0.1" (configmap)
2026-04-11 18:03:18.925537 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="registry.k8s.io/sig-storage/csi-attacher:v4.5.1" (configmap)
2026-04-11 18:03:18.925556 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="registry.k8s.io/sig-storage/csi-snapshotter:v7.0.2" (configmap)
2026-04-11 18:03:18.925602 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="registry.k8s.io/sig-storage/csi-resizer:v1.10.1" (configmap)
2026-04-11 18:03:18.925640 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default)
2026-04-11 18:03:18.925699 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.8.0" (configmap)
2026-04-11 18:03:18.925729 I | op-k8sutil: CSI_TOPOLOGY_DOMAIN_LABELS="" (default)
2026-04-11 18:03:18.925770 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default)
2026-04-11 18:03:18.925801 I | op-k8sutil: ROOK_CSI_NFS_POD_LABELS="" (default)
2026-04-11 18:03:18.925824 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default)
2026-04-11 18:03:18.925860 I | op-k8sutil: CSI_CLUSTER_NAME="" (default)
2026-04-11 18:03:18.925882 I | op-k8sutil: ROOK_CSI_IMAGE_PULL_POLICY="IfNotPresent" (configmap)
2026-04-11 18:03:18.925910 I | op-k8sutil: CSI_CEPHFS_KERNEL_MOUNT_OPTIONS="" (default)
2026-04-11 18:03:18.925946 I | op-k8sutil: CSI_CEPHFS_ATTACH_REQUIRED="true" (configmap)
2026-04-11 18:03:18.925968 I | op-k8sutil: CSI_RBD_ATTACH_REQUIRED="true" (configmap)
2026-04-11 18:03:18.925989 I | op-k8sutil: CSI_NFS_ATTACH_REQUIRED="true" (configmap)
2026-04-11 18:03:18.926033 I | op-k8sutil: CSI_DRIVER_NAME_PREFIX="rook-ceph" (default)
2026-04-11 18:03:18.929809 I | op-k8sutil: CSI_ENABLE_VOLUME_GROUP_SNAPSHOT="true" (configmap)
2026-04-11 18:03:18.929918 I | ceph-csi: skipping csi version check, since unsupported versions are allowed or csi is disabled
2026-04-11 18:03:18.929996 I | ceph-csi: CSI Ceph RBD driver disabled
2026-04-11 18:03:18.930045 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists
2026-04-11 18:03:18.934350 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists
2026-04-11 18:03:19.126282 I | ceph-csi: successfully removed CSI Ceph RBD driver
2026-04-11 18:03:19.126398 I | ceph-csi: CSI CephFS driver disabled
2026-04-11 18:03:19.126460 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists
2026-04-11 18:03:19.129670 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists
2026-04-11 18:03:19.334995 I | ceph-csi: successfully removed CSI CephFS driver
2026-04-11 18:03:19.335139 I | ceph-csi: CSI NFS driver disabled
2026-04-11 18:03:19.335172 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2026-04-11 18:03:19.337816 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2026-04-11 18:03:19.524536 I | ceph-csi: successfully removed CSI NFS driver
2026-04-11 18:03:34.393283 I | ceph-spec: detected ceph image version: "18.2.7-0 reef"
2026-04-11 18:03:34.393313 I | ceph-cluster-controller: validating ceph version from provided image
2026-04-11 18:03:34.400006 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-11 18:03:34.400029 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-11 18:03:34.402510 I | cephclient: writing config file /var/lib/rook/openstack/openstack.config
2026-04-11 18:03:34.402614 I | cephclient: generated admin config in /var/lib/rook/openstack
2026-04-11 18:03:35.058741 E | cephver: external cluster ceph version is a major version higher "18.2.7-0 reef" than the local cluster "0.0.0-0 <unknown version>", consider upgrading
2026-04-11 18:03:35.845298 I | ceph-cluster-controller: cluster "openstack": version "18.2.7-0 reef" detected for image "harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7"
2026-04-11 18:03:36.139976 I | ceph-cluster-controller: creating "rook-config-override" configmap
2026-04-11 18:03:36.142922 I | ceph-cluster-controller: creating "rook-ceph-config" secret
2026-04-11 18:03:36.166647 I | ceph-cluster-controller: external cluster identity established
2026-04-11 18:03:36.166673 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner"
2026-04-11 18:03:37.136685 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node"
2026-04-11 18:03:38.262222 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner"
2026-04-11 18:03:38.977854 I | clusterdisruption-controller: all PGs are active+clean. Restoring default OSD pdb settings
2026-04-11 18:03:38.977903 I | clusterdisruption-controller: creating the default pdb "rook-ceph-osd" with maxUnavailable=1 for all osd
2026-04-11 18:03:38.990680 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:03:39.573863 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node"
2026-04-11 18:03:40.277721 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:03:40.927971 I | ceph-csi: created kubernetes csi secrets for cluster "openstack"
2026-04-11 18:03:40.937391 I | ceph-cluster-controller: successfully updated csi config map
2026-04-11 18:03:40.937420 I | cephclient: getting or creating ceph auth key "client.crash"
2026-04-11 18:03:41.511505 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:03:41.885494 I | ceph-nodedaemon-controller: created kubernetes crash collector secret for cluster "openstack"
2026-04-11 18:03:41.889757 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "openstack"
2026-04-11 18:03:41.889777 I | ceph-cluster-controller: ceph status check interval is 1m0s
2026-04-11 18:03:41.889782 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "openstack"
2026-04-11 18:03:47.798654 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-11 18:03:47.798751 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-11 18:03:47.798771 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7...
2026-04-11 18:03:49.817631 I | ceph-spec: detected ceph image version: "18.2.7-0 reef"
2026-04-11 18:03:51.371089 I | ceph-object-controller: reconciling object store deployments
2026-04-11 18:03:51.406294 I | ceph-object-controller: ceph object store gateway service running at 10.107.152.190
2026-04-11 18:03:51.406349 I | ceph-object-controller: reconciling object store pools
2026-04-11 18:03:54.153229 I | cephclient: reconciling replicated pool ceph.rgw.control succeeded
2026-04-11 18:03:55.091849 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.control"
2026-04-11 18:03:55.091891 I | cephclient: updating pool "ceph.rgw.control" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.control_osd"
2026-04-11 18:03:55.091899 I | cephclient: crush rule "ceph.rgw.control" will no longer be used by pool "ceph.rgw.control"
2026-04-11 18:03:57.234162 I | cephclient: Successfully updated pool "ceph.rgw.control" failure domain to "osd"
2026-04-11 18:03:57.234225 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.control"
2026-04-11 18:04:01.668145 I | cephclient: reconciling replicated pool ceph.rgw.meta succeeded
2026-04-11 18:04:02.797766 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.meta"
2026-04-11 18:04:02.797825 I | cephclient: updating pool "ceph.rgw.meta" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.meta_osd"
2026-04-11 18:04:02.797834 I | cephclient: crush rule "ceph.rgw.meta" will no longer be used by pool "ceph.rgw.meta"
2026-04-11 18:04:04.737324 I | cephclient: Successfully updated pool "ceph.rgw.meta" failure domain to "osd"
2026-04-11 18:04:04.737366 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.meta"
2026-04-11 18:04:08.799184 I | cephclient: reconciling replicated pool ceph.rgw.log succeeded
2026-04-11 18:04:10.206833 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:22} {StateName:creating+peering Count:3}]"
2026-04-11 18:04:10.262842 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:04:10.492511 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.log"
2026-04-11 18:04:10.492544 I | cephclient: updating pool "ceph.rgw.log" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.log_osd"
2026-04-11 18:04:10.492550 I | cephclient: crush rule "ceph.rgw.log" will no longer be used by pool "ceph.rgw.log"
2026-04-11 18:04:11.527229 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:22} {StateName:creating+peering Count:3}]"
2026-04-11 18:04:11.532886 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:04:12.862436 I | cephclient: Successfully updated pool "ceph.rgw.log" failure domain to "osd"
2026-04-11 18:04:12.862472 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.log"
2026-04-11 18:04:16.954293 I | cephclient: reconciling replicated pool ceph.rgw.buckets.index succeeded
2026-04-11 18:04:17.907715 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.index"
2026-04-11 18:04:17.907756 I | cephclient: updating pool "ceph.rgw.buckets.index" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.index_osd"
2026-04-11 18:04:17.907765 I | cephclient: crush rule "ceph.rgw.buckets.index" will no longer be used by pool "ceph.rgw.buckets.index"
2026-04-11 18:04:20.006215 I | cephclient: Successfully updated pool "ceph.rgw.buckets.index" failure domain to "osd"
2026-04-11 18:04:20.006259 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.index"
2026-04-11 18:04:24.082683 I | cephclient: reconciling replicated pool ceph.rgw.buckets.non-ec succeeded
2026-04-11 18:04:25.047478 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.non-ec"
2026-04-11 18:04:25.047519 I | cephclient: updating pool "ceph.rgw.buckets.non-ec" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.non-ec_osd"
2026-04-11 18:04:25.047529 I | cephclient: crush rule "ceph.rgw.buckets.non-ec" will no longer be used by pool "ceph.rgw.buckets.non-ec"
2026-04-11 18:04:27.362193 I | cephclient: Successfully updated pool "ceph.rgw.buckets.non-ec" failure domain to "osd"
2026-04-11 18:04:27.362238 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.non-ec"
2026-04-11 18:04:28.656123 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-11 18:04:32.400491 I | cephclient: reconciling replicated pool ceph.rgw.otp succeeded
2026-04-11 18:04:33.300944 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.otp"
2026-04-11 18:04:33.301016 I | cephclient: updating pool "ceph.rgw.otp" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.otp_osd"
2026-04-11 18:04:33.301024 I | cephclient: crush rule "ceph.rgw.otp" will no longer be used by pool "ceph.rgw.otp"
2026-04-11 18:04:35.454993 I | cephclient: Successfully updated pool "ceph.rgw.otp" failure domain to "osd"
2026-04-11 18:04:35.455036 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.otp"
2026-04-11 18:04:39.533202 I | cephclient: reconciling replicated pool .rgw.root succeeded
2026-04-11 18:04:40.672433 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule ".rgw.root"
2026-04-11 18:04:40.672462 I | cephclient: updating pool ".rgw.root" failure domain from "osd" to "osd" with new crush rule ".rgw.root_osd"
2026-04-11 18:04:40.672466 I | cephclient: crush rule ".rgw.root" will no longer be used by pool ".rgw.root"
2026-04-11 18:04:41.605418 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:49} {StateName:creating+peering Count:7} {StateName:unknown Count:1}]"
2026-04-11 18:04:41.610120 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:04:42.211750 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:49} {StateName:creating+peering Count:7} {StateName:unknown Count:1}]"
2026-04-11 18:04:42.216322 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:04:43.597361 I | cephclient: Successfully updated pool ".rgw.root" failure domain to "osd"
2026-04-11 18:04:43.597405 I | cephclient: setting pool property "pg_num_min" to "8" on pool ".rgw.root"
2026-04-11 18:04:48.679408 I | cephclient: reconciling replicated pool ceph.rgw.buckets.data succeeded
2026-04-11 18:04:49.565331 I | cephclient: creating a new crush rule for changed deviceClass ("default"-->"") on crush rule "ceph.rgw.buckets.data"
2026-04-11 18:04:49.565377 I | cephclient: updating pool "ceph.rgw.buckets.data" failure domain from "osd" to "osd" with new crush rule "ceph.rgw.buckets.data_osd"
2026-04-11 18:04:49.565386 I | cephclient: crush rule "ceph.rgw.buckets.data" will no longer be used by pool "ceph.rgw.buckets.data"
2026-04-11 18:04:52.273327 I | cephclient: Successfully updated pool "ceph.rgw.buckets.data" failure domain to "osd"
2026-04-11 18:04:52.273375 I | ceph-object-controller: configuring object store "ceph"
2026-04-11 18:04:52.927211 I | ceph-object-controller: Object store "ceph": realm=ceph, zonegroup=ceph, zone=ceph
2026-04-11 18:04:53.138321 I | ceph-object-controller: committing changes to RGW configuration period for CephObjectStore "openstack/ceph"
2026-04-11 18:04:53.620911 I | ceph-object-controller: configuration for object-store ceph is complete
2026-04-11 18:04:53.620947 I | ceph-object-controller: creating object store "ceph" in namespace "openstack"
2026-04-11 18:04:53.625035 I | cephclient: getting or creating ceph auth key "client.rgw.ceph.a"
2026-04-11 18:04:54.244931 I | ceph-object-controller: setting rgw config flags
2026-04-11 18:04:54.244974 I | op-config: setting "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-11 18:04:54.721811 I | op-config: successfully set "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-11 18:04:54.721850 I | op-config: setting "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-11 18:04:55.209257 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-11 18:04:55.209314 I | op-config: setting "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-11 18:04:55.696114 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-11 18:04:55.696196 I | op-config: setting "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-11 18:04:56.191422 I | op-config: successfully set "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-11 18:04:56.191471 I | op-config: setting "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-11 18:04:56.673852 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-11 18:04:56.673893 I | op-config: setting "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-11 18:04:57.123900 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-11 18:04:57.124206 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" created
2026-04-11 18:04:57.172352 I | ceph-object-controller: enabling rgw dashboard
2026-04-11 18:04:59.170032 I | ceph-object-controller: created object store "ceph" in namespace "openstack"
2026-04-11 18:04:59.171687 I | ceph-object-controller: setting the dashboard api secret key
2026-04-11 18:04:59.813728 I | ceph-object-controller: done setting the dashboard api secret key
2026-04-11 18:05:12.214967 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:05:12.783728 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:05:14.798193 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-11 18:05:29.605794 E | ceph-object-controller: failed to reconcile CephObjectStore "openstack/ceph". failed to create object store deployments: failed to get COSI user "cosi": Get "http://rook-ceph-rgw-ceph.openstack.svc:80/admin/user?format=json&uid=cosi": dial tcp 10.107.152.190:80: i/o timeout
2026-04-11 18:05:29.627516 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-11 18:05:29.627553 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-11 18:05:29.627577 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7...
2026-04-11 18:05:30.228517 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:05:30.843041 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:05:31.870262 I | ceph-spec: detected ceph image version: "18.2.7-0 reef"
2026-04-11 18:05:33.391697 I | ceph-object-controller: reconciling object store deployments
2026-04-11 18:05:33.421084 I | ceph-object-controller: ceph object store gateway service running at 10.107.152.190
2026-04-11 18:05:33.421121 I | ceph-object-controller: reconciling object store pools
2026-04-11 18:05:34.821901 I | cephclient: application "rgw" is already set on pool "ceph.rgw.control"
2026-04-11 18:05:34.821933 I | cephclient: reconciling replicated pool ceph.rgw.control succeeded
2026-04-11 18:05:35.736090 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.control"
2026-04-11 18:05:38.255096 I | cephclient: application "rgw" is already set on pool "ceph.rgw.meta"
2026-04-11 18:05:38.255133 I | cephclient: reconciling replicated pool ceph.rgw.meta succeeded
2026-04-11 18:05:39.209978 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.meta"
2026-04-11 18:05:41.307275 I | cephclient: application "rgw" is already set on pool "ceph.rgw.log"
2026-04-11 18:05:41.307313 I | cephclient: reconciling replicated pool ceph.rgw.log succeeded
2026-04-11 18:05:42.286744 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.log"
2026-04-11 18:05:43.360653 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:05:43.993596 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:05:46.180187 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.index"
2026-04-11 18:05:46.180225 I | cephclient: reconciling replicated pool ceph.rgw.buckets.index succeeded
2026-04-11 18:05:47.720496 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.index"
2026-04-11 18:05:50.395640 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.non-ec"
2026-04-11 18:05:50.395684 I | cephclient: reconciling replicated pool ceph.rgw.buckets.non-ec succeeded
2026-04-11 18:05:51.307979 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.non-ec"
2026-04-11 18:05:53.459855 I | cephclient: application "rgw" is already set on pool "ceph.rgw.otp"
2026-04-11 18:05:53.459895 I | cephclient: reconciling replicated pool ceph.rgw.otp succeeded
2026-04-11 18:05:54.403414 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.otp"
2026-04-11 18:05:56.434165 I | cephclient: application "rgw" is already set on pool ".rgw.root"
2026-04-11 18:05:56.434205 I | cephclient: reconciling replicated pool .rgw.root succeeded
2026-04-11 18:05:57.338963 I | cephclient: setting pool property "pg_num_min" to "8" on pool ".rgw.root"
2026-04-11 18:05:59.469419 I | cephclient: application "rgw" is already set on pool "ceph.rgw.buckets.data"
2026-04-11 18:05:59.469457 I | cephclient: reconciling replicated pool ceph.rgw.buckets.data succeeded
2026-04-11 18:06:00.988715 I | ceph-object-controller: configuring object store "ceph"
2026-04-11 18:06:01.488396 I | ceph-object-controller: Object store "ceph": realm=ceph, zonegroup=ceph, zone=ceph
2026-04-11 18:06:01.877488 I | ceph-object-controller: there are no changes to commit for RGW configuration period for CephObjectStore "openstack/ceph"
2026-04-11 18:06:01.877525 I | ceph-object-controller: configuration for object-store ceph is complete
2026-04-11 18:06:01.877533 I | ceph-object-controller: creating object store "ceph" in namespace "openstack"
2026-04-11 18:06:01.882083 I | cephclient: getting or creating ceph auth key "client.rgw.ceph.a"
2026-04-11 18:06:02.213880 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-11 18:06:02.659367 I | ceph-object-controller: setting rgw config flags
2026-04-11 18:06:02.659407 I | op-config: setting "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-11 18:06:03.130721 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zone"="ceph" option to the mon configuration database
2026-04-11 18:06:03.130763 I | op-config: setting "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-11 18:06:03.578107 I | op-config: successfully set "client.rgw.ceph.a"="rgw_zonegroup"="ceph" option to the mon configuration database
2026-04-11 18:06:03.578153 I | op-config: setting "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-11 18:06:04.046621 I | op-config: successfully set "client.rgw.ceph.a"="rgw_run_sync_thread"="true" option to the mon configuration database
2026-04-11 18:06:04.046673 I | op-config: setting "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-11 18:06:04.505982 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_nonexistent_bucket"="true" option to the mon configuration database
2026-04-11 18:06:04.506036 I | op-config: setting "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-11 18:06:04.985256 I | op-config: successfully set "client.rgw.ceph.a"="rgw_log_object_name_utc"="true" option to the mon configuration database
2026-04-11 18:06:04.985295 I | op-config: setting "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-11 18:06:05.472931 I | op-config: successfully set "client.rgw.ceph.a"="rgw_enable_usage_log"="true" option to the mon configuration database
2026-04-11 18:06:05.473177 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" created
2026-04-11 18:06:05.492999 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" already exists. updating if needed
2026-04-11 18:06:05.505532 I | op-k8sutil: deployment "rook-ceph-rgw-ceph-a" did not change, nothing to update
2026-04-11 18:06:05.508286 I | ceph-object-controller: config map "rook-ceph-rgw-ceph-mime-types" for object store "ceph" already exists, not overwriting
2026-04-11 18:06:05.512617 I | ceph-object-controller: enabling rgw dashboard
2026-04-11 18:06:06.741176 I | ceph-object-controller: created object store "ceph" in namespace "openstack"
2026-04-11 18:06:07.716715 I | ceph-object-controller: creating COSI user "cosi"
2026-04-11 18:06:07.822191 I | ceph-spec: created ceph *v1.Secret object "rook-ceph-object-user-ceph-cosi"
2026-04-11 18:06:08.427141 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:06:13.933641 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:06:14.578913 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:06:44.512905 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:06:45.186852 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:06:49.689199 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-11 18:07:15.093687 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:07:15.792683 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:07:35.869638 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-11 18:07:45.664789 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:07:46.351357 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:08:16.236311 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:08:16.952755 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:08:22.035159 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-11 18:08:46.792235 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:08:47.535314 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:09:08.200756 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-11 18:09:17.379125 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:09:18.124998 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:09:47.954869 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:09:48.707280 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:09:54.566750 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-11 18:10:18.526325 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:10:19.286043 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:10:40.728409 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-11 18:10:49.120090 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:10:49.883844 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:11:19.713301 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:11:20.486558 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:11:26.863778 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-11 18:11:50.306250 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:11:51.062137 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:12:13.048177 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-11 18:12:20.893437 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:12:21.636590 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:12:51.469078 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:12:52.222396 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:12:59.184790 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-11 18:13:22.042585 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:13:22.801315 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:13:45.347014 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-11 18:13:52.634186 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:13:53.406076 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:14:23.235029 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:14:23.978515 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:14:31.511685 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-11 18:14:53.816672 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-11 18:14:54.577303 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0