I0217 16:13:46.669018 1 cephcsi.go:191] Driver version: v3.11.0 and Git version: bc24b5eca87626d690a29effa9d7420cc0154a7a I0217 16:13:46.669121 1 cephcsi.go:223] Starting driver type: controller with name: rbd.csi.ceph.com I0217 16:13:46.670072 1 leaderelection.go:250] attempting to acquire leader lease kube-system/rbd.csi.ceph.com-kube-system... I0217 16:13:46.702759 1 leaderelection.go:260] successfully acquired lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:13:46.709050 1 reflector.go:289] Starting reflector *v1.PersistentVolume (9h20m24.025180489s) from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:105 I0217 16:13:46.709066 1 reflector.go:325] Listing and watching *v1.PersistentVolume from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:105 I0217 16:13:46.710917 1 reflector.go:351] Caches populated for *v1.PersistentVolume from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:105 I0217 16:13:46.723382 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:13:48.849461 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:13:50.855049 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:13:52.862970 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:13:54.871071 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:13:56.882043 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:13:58.891755 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:00.901236 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:02.911941 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:04.918735 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:06.925399 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:09.035164 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:11.045067 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:13.052962 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:15.061027 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:17.069613 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:19.081963 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:21.090577 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:23.100443 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:25.192169 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:27.221824 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:29.233033 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:31.308840 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:33.317624 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:35.327737 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:37.338885 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:39.347899 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:41.357544 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:43.368310 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:45.377539 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:47.388323 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:49.398905 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:51.406060 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:53.415382 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:55.427060 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:57.437167 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:14:59.479214 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:01.488931 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:03.514160 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:05.526318 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:07.540553 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:09.553500 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:11.564574 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:16.018160 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:18.031650 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:20.055037 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:22.068048 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:24.075577 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:26.183024 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:28.203319 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:30.213475 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:32.229632 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:34.319905 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:36.331407 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:38.403886 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:40.443402 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:42.455607 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:44.486620 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:46.497522 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:48.515880 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:50.531589 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:52.717005 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:54.809607 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:56.897472 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:15:58.940260 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:01.113155 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:03.124925 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:05.150533 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:07.210188 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:09.583626 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:11.601931 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:13.615704 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:15.637483 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:17.652112 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:19.658962 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:21.783490 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:23.792993 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:25.804003 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:27.812062 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:29.831293 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:31.841324 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:33.861001 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:35.874298 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:37.950898 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:39.960642 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:41.974840 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:44.091889 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:46.101081 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:48.165263 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:50.192468 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:52.199853 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:54.246400 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:16:59.294421 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:01.965733 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:04.620547 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:06.751498 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:08.868109 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:10.898046 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:12.907830 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:15.111906 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:17.191313 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:19.241193 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system [controller-runtime] log.SetLogger(...) was never called; logs will not be displayed. Detected at: > goroutine 88 [running]: > runtime/debug.Stack() > /usr/local/go/src/runtime/debug/stack.go:24 +0x5e > sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot() > /go/src/github.com/ceph/ceph-csi/vendor/sigs.k8s.io/controller-runtime/pkg/log/log.go:60 +0xcd > sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).WithValues(0xc00082d700, {0xc000991320, 0x6, 0x6}) > /go/src/github.com/ceph/ceph-csi/vendor/sigs.k8s.io/controller-runtime/pkg/log/deleg.go:168 +0x49 > github.com/go-logr/logr.Logger.WithValues(...) > /go/src/github.com/ceph/ceph-csi/vendor/github.com/go-logr/logr/logr.go:332 > sigs.k8s.io/controller-runtime/pkg/controller.NewUnmanaged.func1(0xc0003e45c0) > /go/src/github.com/ceph/ceph-csi/vendor/sigs.k8s.io/controller-runtime/pkg/controller/controller.go:121 +0x1d7 > sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0004bc280, {0x27c0488, 0xc0008441e0}, {0x1f93660?, 0xc0003e45a0?}) > /go/src/github.com/ceph/ceph-csi/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:306 +0x16a > sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0004bc280, {0x27c0488, 0xc0008441e0}) > /go/src/github.com/ceph/ceph-csi/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266 +0x1af > sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() > /go/src/github.com/ceph/ceph-csi/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227 +0x79 > created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 75 > /go/src/github.com/ceph/ceph-csi/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:223 +0x565 I0217 16:17:20.895530 1 reflector.go:289] Starting reflector *v1.Secret (9h11m32.381587281s) from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:105 I0217 16:17:20.895565 1 reflector.go:325] Listing and watching *v1.Secret from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:105 I0217 16:17:20.930511 1 reflector.go:351] Caches populated for *v1.Secret from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:105 I0217 16:17:21.021819 1 omap.go:89] got omap values: (pool="kube", namespace="", name="csi.volumes.default"): map[csi.volume.pvc-36eba49b-2a85-4ff9-9c5f-a0348a58ac10:5ddd9360-2435-4b90-8421-858091ccad1d] I0217 16:17:21.023567 1 omap.go:89] got omap values: (pool="kube", namespace="", name="csi.volume.5ddd9360-2435-4b90-8421-858091ccad1d"): map[csi.imageid:37d9c02b2603 csi.imagename:csi-vol-5ddd9360-2435-4b90-8421-858091ccad1d csi.volname:pvc-36eba49b-2a85-4ff9-9c5f-a0348a58ac10 csi.volume.owner:default] I0217 16:17:21.100488 1 omap.go:89] got omap values: (pool="kube", namespace="", name="csi.volumes.default"): map[csi.volume.pvc-36eba49b-2a85-4ff9-9c5f-a0348a58ac10:5ddd9360-2435-4b90-8421-858091ccad1d] I0217 16:17:21.102229 1 omap.go:89] got omap values: (pool="kube", namespace="", name="csi.volume.5ddd9360-2435-4b90-8421-858091ccad1d"): map[csi.imageid:37d9c02b2603 csi.imagename:csi-vol-5ddd9360-2435-4b90-8421-858091ccad1d csi.volname:pvc-36eba49b-2a85-4ff9-9c5f-a0348a58ac10 csi.volume.owner:default] I0217 16:17:21.254178 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:23.267571 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:25.315319 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:27.327714 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:29.422146 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:31.464655 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:33.712033 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:35.728242 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:37.743829 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:39.755045 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:41.763211 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:43.774025 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:46.451142 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:48.462838 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:50.474359 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:52.489968 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:54.527892 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:57.295547 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:17:59.577780 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:18:01.666026 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:18:03.705039 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:18:05.729159 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:18:07.739268 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:18:09.800988 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:18:11.817145 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:18:13.846158 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:18:15.871042 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:18:17.880603 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:18:19.892095 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:18:22.001880 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:18:24.015451 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0217 16:18:26.034692 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system E0217 16:18:38.366824 1 leaderelection.go:332] error retrieving resource lock kube-system/rbd.csi.ceph.com-kube-system: Get "https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rbd.csi.ceph.com-kube-system": context deadline exceeded I0217 16:18:38.366883 1 leaderelection.go:285] failed to renew lease kube-system/rbd.csi.ceph.com-kube-system: timed out waiting for the condition E0217 16:18:38.366938 1 controller.go:85] failed to start manager leader election lost E0217 16:18:38.366945 1 cephcsi.go:314] leader election lost