2026/04/08 23:12:28 maxprocs: Updating GOMAXPROCS=1: determined from CPU quota
2026-04-08 23:12:28.508723 I | rookcmd: starting Rook v1.15.9 with arguments '/usr/local/bin/rook ceph operator'
2026-04-08 23:12:28.508749 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO
2026-04-08 23:12:28.508751 I | cephcmd: starting Rook-Ceph operator
2026-04-08 23:12:28.625143 I | cephcmd: base ceph version inside the rook operator image is "ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)"
2026-04-08 23:12:28.633667 I | op-k8sutil: operator setting "CSI_CEPHFS_PLUGIN_RESOURCE" = "- name : driver-registrar\n  resource:\n    requests:\n      memory: 128Mi\n      cpu: 50m\n    limits:\n      memory: 256Mi\n- name : csi-cephfsplugin\n  resource:\n    requests:\n      memory: 512Mi\n      cpu: 250m\n    limits:\n      memory: 1Gi\n- name : liveness-prometheus\n  resource:\n    requests:\n      memory: 128Mi\n      cpu: 50m\n    limits:\n      memory: 256Mi\n"
2026-04-08 23:12:28.633687 I | op-k8sutil: operator setting "CSI_ENABLE_CEPHFS_SNAPSHOTTER" = "true"
2026-04-08 23:12:28.633699 I | op-k8sutil: operator setting "CSI_ENABLE_OMAP_GENERATOR" = "false"
2026-04-08 23:12:28.633705 I | op-k8sutil: operator setting "CSI_ENABLE_RBD_SNAPSHOTTER" = "true"
2026-04-08 23:12:28.633709 I | op-k8sutil: operator setting "CSI_FORCE_CEPHFS_KERNEL_CLIENT" = "true"
2026-04-08 23:12:28.633715 I | op-k8sutil: operator setting "ROOK_CSI_SNAPSHOTTER_IMAGE" = "registry.k8s.io/sig-storage/csi-snapshotter:v8.0.1"
2026-04-08 23:12:28.633719 I | op-k8sutil: operator setting "CSI_CEPHFS_FSGROUPPOLICY" = "File"
2026-04-08 23:12:28.633723 I | op-k8sutil: operator setting "CSI_RBD_ATTACH_REQUIRED" = "true"
2026-04-08 23:12:28.633728 I | op-k8sutil: operator setting "ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS" = "15"
2026-04-08 23:12:28.633733 I | op-k8sutil: operator setting "ROOK_CSI_DISABLE_DRIVER" = "false"
2026-04-08 23:12:28.633739 I | op-k8sutil: operator setting "ROOK_CSI_RESIZER_IMAGE" = "registry.k8s.io/sig-storage/csi-resizer:v1.11.1"
2026-04-08 23:12:28.633744 I | op-k8sutil: operator setting "CSI_ENABLE_ENCRYPTION" = "false"
2026-04-08 23:12:28.633748 I | op-k8sutil: operator setting "CSI_ENABLE_TOPOLOGY" = "false"
2026-04-08 23:12:28.633760 I | op-k8sutil: operator setting "CSI_NFS_PLUGIN_RESOURCE" = "- name : driver-registrar\n  resource:\n    requests:\n      memory: 128Mi\n      cpu: 50m\n    limits:\n      memory: 256Mi\n- name : csi-nfsplugin\n  resource:\n    requests:\n      memory: 512Mi\n      cpu: 250m\n    limits:\n      memory: 1Gi\n"
2026-04-08 23:12:28.633771 I | op-k8sutil: operator setting "CSI_PROVISIONER_PRIORITY_CLASSNAME" = "system-cluster-critical"
2026-04-08 23:12:28.633776 I | op-k8sutil: operator setting "CSI_RBD_FSGROUPPOLICY" = "File"
2026-04-08 23:12:28.633794 I | op-k8sutil: operator setting "CSI_RBD_PROVISIONER_RESOURCE" = "- name : csi-provisioner\n  resource:\n    requests:\n      memory: 128Mi\n      cpu: 100m\n    limits:\n      memory: 256Mi\n- name : csi-resizer\n  resource:\n    requests:\n      memory: 128Mi\n      cpu: 100m\n    limits:\n      memory: 256Mi\n- name : csi-attacher\n  resource:\n    requests:\n      memory: 128Mi\n      cpu: 100m\n    limits:\n      memory: 256Mi\n- name : csi-snapshotter\n  resource:\n    requests:\n      memory: 128Mi\n      cpu: 100m\n    limits:\n      memory: 256Mi\n- name : csi-rbdplugin\n  resource:\n    requests:\n      memory: 512Mi\n    limits:\n      memory: 1Gi\n- name : csi-omap-generator\n  resource:\n    requests:\n      memory: 512Mi\n      cpu: 250m\n    limits:\n      memory: 1Gi\n- name : liveness-prometheus\n  resource:\n    requests:\n      memory: 128Mi\n      cpu: 50m\n    limits:\n      memory: 256Mi\n"
2026-04-08 23:12:28.633800 I | op-k8sutil: operator setting "ROOK_CSI_REGISTRAR_IMAGE" = "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.11.1"
2026-04-08 23:12:28.633814 I | op-k8sutil: operator setting "CSI_CEPHFS_PROVISIONER_RESOURCE" = "- name : csi-provisioner\n  resource:\n    requests:\n      memory: 128Mi\n      cpu: 100m\n    limits:\n      memory: 256Mi\n- name : csi-resizer\n  resource:\n    requests:\n      memory: 128Mi\n      cpu: 100m\n    limits:\n      memory: 256Mi\n- name : csi-attacher\n  resource:\n    requests:\n      memory: 128Mi\n      cpu: 100m\n    limits:\n      memory: 256Mi\n- name : csi-snapshotter\n  resource:\n    requests:\n      memory: 128Mi\n      cpu: 100m\n    limits:\n      memory: 256Mi\n- name : csi-cephfsplugin\n  resource:\n    requests:\n      memory: 512Mi\n      cpu: 250m\n    limits:\n      memory: 1Gi\n- name : liveness-prometheus\n  resource:\n    requests:\n      memory: 128Mi\n      cpu: 50m\n    limits:\n      memory: 256Mi\n"
2026-04-08 23:12:28.633828 I | op-k8sutil: operator setting "CSI_DISABLE_HOLDER_PODS" = "true"
2026-04-08 23:12:28.633833 I | op-k8sutil: operator setting "CSI_PLUGIN_PRIORITY_CLASSNAME" = "system-node-critical"
2026-04-08 23:12:28.633838 I | op-k8sutil: operator setting "CSI_PROVISIONER_REPLICAS" = "2"
2026-04-08 23:12:28.633842 I | op-k8sutil: operator setting "ROOK_CEPH_ALLOW_LOOP_DEVICES" = "false"
2026-04-08 23:12:28.633846 I | op-k8sutil: operator setting "ROOK_CSIADDONS_IMAGE" = "quay.io/csiaddons/k8s-sidecar:v0.9.1"
2026-04-08 23:12:28.633851 I | op-k8sutil: operator setting "ROOK_CSI_ENABLE_NFS" = "false"
2026-04-08 23:12:28.633855 I | op-k8sutil: operator setting "CSI_ENABLE_NFS_SNAPSHOTTER" = "true"
2026-04-08 23:12:28.633859 I | op-k8sutil: operator setting "CSI_NFS_FSGROUPPOLICY" = "File"
2026-04-08 23:12:28.633870 I | op-k8sutil: operator setting "CSI_NFS_PROVISIONER_RESOURCE" = "- name : csi-provisioner\n  resource:\n    requests:\n      memory: 128Mi\n      cpu: 100m\n    limits:\n      memory: 256Mi\n- name : csi-nfsplugin\n  resource:\n    requests:\n      memory: 512Mi\n      cpu: 250m\n    limits:\n      memory: 1Gi\n- name : csi-attacher\n  resource:\n    requests:\n      memory: 512Mi\n      cpu: 250m\n    limits:\n      memory: 1Gi\n"
2026-04-08 23:12:28.633881 I | op-k8sutil: operator setting "ROOK_LOG_LEVEL" = "INFO"
2026-04-08 23:12:28.633886 I | op-k8sutil: operator setting "CSI_ENABLE_CSIADDONS" = "false"
2026-04-08 23:12:28.633891 I | op-k8sutil: operator setting "CSI_ENABLE_METADATA" = "false"
2026-04-08 23:12:28.633896 I | op-k8sutil: operator setting "CSI_ENABLE_VOLUME_GROUP_SNAPSHOT" = "true"
2026-04-08 23:12:28.633900 I | op-k8sutil: operator setting "CSI_GRPC_TIMEOUT_SECONDS" = "150"
2026-04-08 23:12:28.633906 I | op-k8sutil: operator setting "ROOK_CSI_ATTACHER_IMAGE" = "registry.k8s.io/sig-storage/csi-attacher:v4.6.1"
2026-04-08 23:12:28.633910 I | op-k8sutil: operator setting "ROOK_CSI_PROVISIONER_IMAGE" = "registry.k8s.io/sig-storage/csi-provisioner:v5.0.1"
2026-04-08 23:12:28.633914 I | op-k8sutil: operator setting "CSI_ENABLE_HOST_NETWORK" = "true"
2026-04-08 23:12:28.633918 I | op-k8sutil: operator setting "ROOK_CSI_ENABLE_RBD" = "false"
2026-04-08 23:12:28.633924 I | op-k8sutil: operator setting "ROOK_ENABLE_DISCOVERY_DAEMON" = "false"
2026-04-08 23:12:28.633928 I | op-k8sutil: operator setting "ROOK_OBC_WATCH_OPERATOR_NAMESPACE" = "true"
2026-04-08 23:12:28.633932 I | op-k8sutil: operator setting "CSI_CEPHFS_ATTACH_REQUIRED" = "true"
2026-04-08 23:12:28.633935 I | op-k8sutil: operator setting "CSI_NFS_ATTACH_REQUIRED" = "true"
2026-04-08 23:12:28.633943 I | op-k8sutil: operator setting "CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT" = "false"
2026-04-08 23:12:28.633951 I | op-k8sutil: operator setting "CSI_RBD_PLUGIN_RESOURCE" = "- name : driver-registrar\n  resource:\n    requests:\n      memory: 128Mi\n      cpu: 50m\n    limits:\n      memory: 256Mi\n- name : csi-rbdplugin\n  resource:\n    requests:\n      memory: 512Mi\n      cpu: 250m\n    limits:\n      memory: 1Gi\n- name : liveness-prometheus\n  resource:\n    requests:\n      memory: 128Mi\n      cpu: 50m\n    limits:\n      memory: 256Mi\n"
2026-04-08 23:12:28.633956 I | op-k8sutil: operator setting "ROOK_CSI_CEPH_IMAGE" = "quay.io/cephcsi/cephcsi:v3.12.3"
2026-04-08 23:12:28.633962 I | op-k8sutil: operator setting "ROOK_CSI_ENABLE_CEPHFS" = "false"
2026-04-08 23:12:28.633966 I | op-k8sutil: operator setting "ROOK_CSI_IMAGE_PULL_POLICY" = "IfNotPresent"
2026-04-08 23:12:28.633972 I | operator: watching all namespaces for Ceph CRs
2026-04-08 23:12:28.634065 I | operator: setting up schemes
2026-04-08 23:12:28.637021 I | operator: setting up the controller-runtime manager
2026-04-08 23:12:28.637506 I | ceph-cluster-controller: successfully started
2026-04-08 23:12:28.637953 I | ceph-cluster-controller: enabling hotplug orchestration
2026-04-08 23:12:28.637977 I | ceph-nodedaemon-controller: successfully started
2026-04-08 23:12:28.637990 I | ceph-block-pool-controller: successfully started
2026-04-08 23:12:28.638004 I | ceph-object-store-user-controller: successfully started
2026-04-08 23:12:28.638021 I | ceph-object-realm-controller: successfully started
2026-04-08 23:12:28.638036 I | ceph-object-zonegroup-controller: successfully started
2026-04-08 23:12:28.638046 I | ceph-object-zone-controller: successfully started
2026-04-08 23:12:28.638133 I | ceph-object-controller: successfully started
2026-04-08 23:12:28.638170 I | ceph-file-controller: successfully started
2026-04-08 23:12:28.638191 I | ceph-nfs-controller: successfully started
2026-04-08 23:12:28.638220 I | ceph-rbd-mirror-controller: successfully started
2026-04-08 23:12:28.638246 I | ceph-client-controller: successfully started
2026-04-08 23:12:28.638260 I | ceph-filesystem-mirror-controller: successfully started
2026-04-08 23:12:28.638282 I | operator: rook-ceph-operator-config-controller successfully started
2026-04-08 23:12:28.638296 I | ceph-csi: rook-ceph-operator-csi-controller successfully started
2026-04-08 23:12:28.638439 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started
2026-04-08 23:12:28.638457 I | ceph-bucket-topic: successfully started
2026-04-08 23:12:28.638464 I | ceph-bucket-notification: successfully started
2026-04-08 23:12:28.638471 I | ceph-bucket-notification: successfully started
2026-04-08 23:12:28.638479 I | ceph-fs-subvolumegroup-controller: successfully started
2026-04-08 23:12:28.638525 I | blockpool-rados-namespace-controller: successfully started
2026-04-08 23:12:28.638578 I | ceph-cosi-controller: successfully started
2026-04-08 23:12:28.638596 I | operator: starting the controller-runtime manager
2026-04-08 23:12:29.081378 I | operator: rook-ceph-operator-config-controller done reconciling
2026-04-08 23:12:29.130358 I | ceph-csi: successfully created csi config map "rook-ceph-csi-config"
2026-04-08 23:12:29.134574 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists
2026-04-08 23:12:29.136922 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists
2026-04-08 23:12:29.143516 I | ceph-csi: successfully removed CSI Ceph RBD driver
2026-04-08 23:12:29.143531 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists
2026-04-08 23:12:29.146093 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists
2026-04-08 23:12:29.152020 I | ceph-csi: successfully removed CSI CephFS driver
2026-04-08 23:12:29.152040 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2026-04-08 23:12:29.155064 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2026-04-08 23:12:29.161422 I | ceph-csi: successfully removed CSI NFS driver
2026-04-08 23:12:51.378042 I | ceph-spec: adding finalizer "cephcluster.ceph.rook.io" on "ceph"
2026-04-08 23:12:51.383566 I | ceph-spec: adding finalizer "cephobjectstore.ceph.rook.io" on "ceph"
2026-04-08 23:12:51.385675 I | clusterdisruption-controller: deleted all legacy node drain canary pods
2026-04-08 23:12:51.395111 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-08 23:12:51.395142 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-08 23:12:51.395341 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "openstack.ceph.rook.io/bucket"
2026-04-08 23:12:51.395932 I | op-bucket-prov: successfully reconciled bucket provisioner
I0408 23:12:51.396050       1 manager.go:135] "msg"="starting provisioner" "logger"="objectbucket.io/provisioner-manager" "name"="openstack.ceph.rook.io/bucket"
2026-04-08 23:12:51.397185 I | ceph-cluster-controller: reconciling ceph cluster in namespace "openstack"
2026-04-08 23:12:51.408506 E | ceph-spec: failed to update cluster condition to {Type:Connecting Status:True Reason:ClusterConnecting Message:Attempting to connect to an external Ceph cluster LastHeartbeatTime:2026-04-08 23:12:51.397247949 +0000 UTC m=+22.919912229 LastTransitionTime:2026-04-08 23:12:51.397247859 +0000 UTC m=+22.919912149}. failed to update object "openstack/ceph" status: Operation cannot be fulfilled on cephclusters.ceph.rook.io "ceph": the object has been modified; please apply your changes to the latest version and try again
2026-04-08 23:12:51.416298 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-08 23:12:51.416330 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-08 23:12:51.416362 I | ceph-spec: found the cluster info to connect to the external cluster. will use "client.admin" to check health and monitor status. mons=map[instance:0xc0016b6f30]
2026-04-08 23:12:51.416615 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7...
2026-04-08 23:12:52.385662 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-08 23:12:52.385723 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-08 23:12:52.791061 I | op-k8sutil: removing daemonset csi-rbdplugin if it exists
2026-04-08 23:12:52.794132 I | op-k8sutil: removing deployment csi-rbdplugin-provisioner if it exists
2026-04-08 23:12:52.987612 I | ceph-csi: successfully removed CSI Ceph RBD driver
2026-04-08 23:12:52.987644 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists
2026-04-08 23:12:52.990803 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists
2026-04-08 23:12:53.186796 I | ceph-csi: successfully removed CSI CephFS driver
2026-04-08 23:12:53.186830 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2026-04-08 23:12:53.188818 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2026-04-08 23:12:53.388189 I | ceph-csi: successfully removed CSI NFS driver
2026-04-08 23:13:08.863765 I | ceph-spec: detected ceph image version: "18.2.7-0 reef"
2026-04-08 23:13:08.863798 I | ceph-cluster-controller: validating ceph version from provided image
2026-04-08 23:13:08.869648 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-08 23:13:08.869682 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-08 23:13:08.873068 I | cephclient: writing config file /var/lib/rook/openstack/openstack.config
2026-04-08 23:13:08.873200 I | cephclient: generated admin config in /var/lib/rook/openstack
2026-04-08 23:13:09.364178 E | cephver: external cluster ceph version is a major version higher "18.2.7-0 reef" than the local cluster "0.0.0-0 <unknown version>", consider upgrading
2026-04-08 23:13:09.861368 I | ceph-cluster-controller: cluster "openstack": version "18.2.7-0 reef" detected for image "harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7"
2026-04-08 23:13:10.055489 I | ceph-cluster-controller: creating "rook-config-override" configmap
2026-04-08 23:13:10.058311 I | ceph-cluster-controller: creating "rook-ceph-config" secret
2026-04-08 23:13:10.075655 I | ceph-cluster-controller: external cluster identity established
2026-04-08 23:13:10.075693 I | cephclient: getting or creating ceph auth key "client.csi-rbd-provisioner"
2026-04-08 23:13:10.657285 I | cephclient: getting or creating ceph auth key "client.csi-rbd-node"
2026-04-08 23:13:11.230557 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-provisioner"
2026-04-08 23:13:11.862055 I | cephclient: getting or creating ceph auth key "client.csi-cephfs-node"
2026-04-08 23:13:12.660905 I | clusterdisruption-controller: all PGs are active+clean. Restoring default OSD pdb settings
2026-04-08 23:13:12.660960 I | clusterdisruption-controller: creating the default pdb "rook-ceph-osd" with maxUnavailable=1 for all osd
2026-04-08 23:13:12.676255 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:13:12.988008 I | ceph-csi: created kubernetes csi secrets for cluster "openstack"
2026-04-08 23:13:12.999618 I | ceph-cluster-controller: successfully updated csi config map
2026-04-08 23:13:12.999654 I | cephclient: getting or creating ceph auth key "client.crash"
2026-04-08 23:13:13.669231 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:13:13.902244 I | ceph-nodedaemon-controller: created kubernetes crash collector secret for cluster "openstack"
2026-04-08 23:13:13.905914 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "openstack"
2026-04-08 23:13:13.905952 I | ceph-cluster-controller: ceph status check interval is 1m0s
2026-04-08 23:13:13.905956 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "openstack"
2026-04-08 23:13:21.474896 I | ceph-spec: parsing mon endpoints: instance=10.96.240.200:6789
2026-04-08 23:13:21.474929 I | ceph-spec: updating obsolete maxMonID 0 to actual value 76846025234
2026-04-08 23:13:21.474947 I | ceph-spec: detecting the ceph image version for image harbor.atmosphere.dev/quay.io/ceph/ceph:v18.2.7...
2026-04-08 23:13:23.670393 I | ceph-spec: detected ceph image version: "18.2.7-0 reef"
2026-04-08 23:13:24.939091 I | ceph-object-controller: reconciling object store deployments
2026-04-08 23:13:24.956919 I | ceph-object-controller: ceph object store gateway service running at 10.101.102.34
2026-04-08 23:13:24.956947 I | ceph-object-controller: reconciling object store pools
2026-04-08 23:13:27.873856 I | cephclient: reconciling replicated pool ceph.rgw.control succeeded
2026-04-08 23:13:27.873912 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.control"
2026-04-08 23:13:32.050901 I | cephclient: reconciling replicated pool ceph.rgw.meta succeeded
2026-04-08 23:13:32.050945 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.meta"
2026-04-08 23:13:36.103429 I | cephclient: reconciling replicated pool ceph.rgw.log succeeded
2026-04-08 23:13:36.103474 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.log"
2026-04-08 23:13:40.175894 I | cephclient: reconciling replicated pool ceph.rgw.buckets.index succeeded
2026-04-08 23:13:40.175927 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.index"
2026-04-08 23:13:43.509329 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:28} {StateName:unknown Count:5}]"
2026-04-08 23:13:43.514846 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:13:44.129694 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:28} {StateName:unknown Count:5}]"
2026-04-08 23:13:44.134474 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:13:45.260025 I | cephclient: reconciling replicated pool ceph.rgw.buckets.non-ec succeeded
2026-04-08 23:13:45.260062 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.buckets.non-ec"
2026-04-08 23:13:49.328621 I | cephclient: reconciling replicated pool ceph.rgw.otp succeeded
2026-04-08 23:13:49.328677 I | cephclient: setting pool property "pg_num_min" to "8" on pool "ceph.rgw.otp"
2026-04-08 23:13:53.406719 I | cephclient: reconciling replicated pool .rgw.root succeeded
2026-04-08 23:13:53.406756 I | cephclient: setting pool property "pg_num_min" to "8" on pool ".rgw.root"
2026-04-08 23:13:58.252914 I | cephclient: reconciling replicated pool ceph.rgw.buckets.data succeeded
2026-04-08 23:13:58.252947 I | ceph-object-controller: configuring object store "ceph"
2026-04-08 23:13:59.079244 I | ceph-object-controller: Object store "ceph": realm=ceph, zonegroup=ceph, zone=ceph
2026-04-08 23:13:59.472188 I | ceph-object-controller: committing changes to RGW configuration period for CephObjectStore "openstack/ceph"
2026-04-08 23:13:59.981237 I | ceph-object-controller: configuration for object-store ceph is complete
2026-04-08 23:13:59.985655 I | ceph-object-controller: creating object store "ceph" in namespace "openstack"
2026-04-08 23:13:59.990358 I | cephclient: getting or creating ceph auth key "client.rgw.ceph.a"
2026-04-08 23:14:00.762784 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-08 23:14:00.901408 I | ceph-object-controller: setting rgw config flags
2026-04-08 23:14:00.901456 I | ceph-object-controller: Configuring authentication with keystone
2026-04-08 23:14:00.901498 I | op-config: setting option "rgw_keystone_accepted_roles" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:01.302497 I | op-config: successfully set option "rgw_keystone_accepted_roles" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:01.302532 I | op-config: setting option "rgw_keystone_admin_user" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:01.734374 I | op-config: successfully set option "rgw_keystone_admin_user" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:01.734418 I | op-config: setting option "rgw_swift_versioning_enabled" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:02.144239 I | op-config: successfully set option "rgw_swift_versioning_enabled" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:02.144279 I | op-config: setting option "rgw_run_sync_thread" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:02.541898 I | op-config: successfully set option "rgw_run_sync_thread" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:02.541935 I | op-config: setting option "rgw_zone" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:02.954903 I | op-config: successfully set option "rgw_zone" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:02.954952 I | op-config: setting option "rgw_zonegroup" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:03.351776 I | op-config: successfully set option "rgw_zonegroup" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:03.351818 I | op-config: setting option "rgw_keystone_implicit_tenants" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:03.759061 I | op-config: successfully set option "rgw_keystone_implicit_tenants" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:03.759105 I | op-config: setting option "rgw_log_nonexistent_bucket" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:04.176011 I | op-config: successfully set option "rgw_log_nonexistent_bucket" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:04.176055 I | op-config: setting option "rgw_enable_usage_log" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:04.571841 I | op-config: successfully set option "rgw_enable_usage_log" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:04.571876 I | op-config: setting option "rgw_keystone_api_version" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:04.998749 I | op-config: successfully set option "rgw_keystone_api_version" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:04.998811 I | op-config: setting option "rgw_keystone_admin_domain" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:05.415766 I | op-config: successfully set option "rgw_keystone_admin_domain" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:05.415809 I | op-config: setting option "rgw_keystone_admin_project" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:05.806892 I | op-config: successfully set option "rgw_keystone_admin_project" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:05.806932 I | op-config: setting option "rgw_log_object_name_utc" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:06.210412 I | op-config: successfully set option "rgw_log_object_name_utc" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:06.210456 I | op-config: setting option "rgw_keystone_admin_password" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:06.612383 I | op-config: successfully set option "rgw_keystone_admin_password" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:06.612429 I | op-config: setting option "rgw_s3_auth_use_keystone" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:07.017787 I | op-config: successfully set option "rgw_s3_auth_use_keystone" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:07.017835 I | op-config: setting option "rgw_swift_account_in_url" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:07.411273 I | op-config: successfully set option "rgw_swift_account_in_url" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:07.411319 I | op-config: setting option "rgw_keystone_url" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:07.838504 I | op-config: successfully set option "rgw_keystone_url" (user "client.rgw.ceph.a") to the mon configuration database
2026-04-08 23:14:07.838875 I | ceph-object-controller: object store "ceph" deployment "rook-ceph-rgw-ceph-a" created
2026-04-08 23:14:07.891217 I | ceph-object-controller: enabling rgw dashboard
2026-04-08 23:14:09.748157 I | ceph-object-controller: created object store "ceph" in namespace "openstack"
2026-04-08 23:14:09.748842 I | ceph-object-controller: setting the dashboard api secret key
2026-04-08 23:14:10.673330 I | ceph-object-controller: done setting the dashboard api secret key
2026-04-08 23:14:10.787137 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:14:14.021434 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:14:14.599236 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:14:44.503822 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:14:45.094066 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:14:46.699300 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-08 23:15:14.980521 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:15:15.608527 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:15:32.677161 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-08 23:15:45.479087 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:15:46.083496 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:16:15.984431 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:16:16.568315 I | clusterdisruption-controller: all "host" failure domains: []. osd is down in failure domain: "". active node drains: false. pg health: "cluster is not fully clean. PGs: [{StateName:active+clean Count:97} {StateName:unknown Count:24}]"
2026-04-08 23:16:16.572515 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:16:18.784122 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-08 23:16:46.450655 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:16:47.043651 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:17:04.728610 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-08 23:17:16.916105 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:17:17.520755 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:17:47.376156 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:17:47.995979 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:17:50.682500 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-08 23:18:17.839291 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:18:18.463919 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:18:36.618551 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-08 23:18:48.316434 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:18:48.933650 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:19:18.787254 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:19:19.397864 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:19:22.596363 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-08 23:19:49.252230 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:19:49.898908 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:20:08.544417 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-08 23:20:19.743690 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:20:20.377461 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:20:50.228994 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:20:50.848743 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:20:54.468999 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-08 23:21:20.755463 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:21:21.346062 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:21:40.436769 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-08 23:21:51.285347 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:21:51.830935 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:22:21.788493 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:22:22.327684 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:22:26.386741 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-08 23:22:52.270244 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:22:52.794379 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:23:12.349152 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-08 23:23:22.753446 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:23:23.265318 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:23:53.250492 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:23:53.741111 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:23:58.307658 W | op-mon: external cluster mon count is 1, consider adding new monitors.
2026-04-08 23:24:23.742098 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0
2026-04-08 23:24:24.224340 I | clusterdisruption-controller: reconciling osd pdb reconciler as the allowed disruptions in default pdb is 0