I0320 10:23:13.224735 1 cephcsi.go:191] Driver version: v3.11.0 and Git version: bc24b5eca87626d690a29effa9d7420cc0154a7a I0320 10:23:13.224943 1 cephcsi.go:223] Starting driver type: controller with name: rbd.csi.ceph.com I0320 10:23:13.225540 1 leaderelection.go:250] attempting to acquire leader lease kube-system/rbd.csi.ceph.com-kube-system... E0320 10:23:43.254522 1 leaderelection.go:332] error retrieving resource lock kube-system/rbd.csi.ceph.com-kube-system: Get "https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rbd.csi.ceph.com-kube-system": dial tcp 10.96.0.1:443: i/o timeout I0320 10:23:43.254582 1 leaderelection.go:255] failed to acquire lease kube-system/rbd.csi.ceph.com-kube-system E0320 10:24:16.036738 1 leaderelection.go:332] error retrieving resource lock kube-system/rbd.csi.ceph.com-kube-system: Get "https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rbd.csi.ceph.com-kube-system": dial tcp 10.96.0.1:443: i/o timeout I0320 10:24:16.036810 1 leaderelection.go:255] failed to acquire lease kube-system/rbd.csi.ceph.com-kube-system E0320 10:24:49.627712 1 leaderelection.go:332] error retrieving resource lock kube-system/rbd.csi.ceph.com-kube-system: Get "https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rbd.csi.ceph.com-kube-system": dial tcp 10.96.0.1:443: i/o timeout I0320 10:24:49.627760 1 leaderelection.go:255] failed to acquire lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:24:56.753261 1 leaderelection.go:354] lock is held by ceph-csi-rbd-provisioner-69cb97fb6-7sf5r_c5b8b501-4629-4d4f-acaa-3174459f35ee and has not yet expired I0320 10:24:56.753285 1 leaderelection.go:255] failed to acquire lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:00.276120 1 leaderelection.go:354] lock is held by ceph-csi-rbd-provisioner-69cb97fb6-7sf5r_c5b8b501-4629-4d4f-acaa-3174459f35ee and has not yet expired I0320 10:25:00.276163 1 leaderelection.go:255] failed to acquire lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:02.374507 1 leaderelection.go:354] lock is held by ceph-csi-rbd-provisioner-69cb97fb6-7sf5r_c5b8b501-4629-4d4f-acaa-3174459f35ee and has not yet expired I0320 10:25:02.374542 1 leaderelection.go:255] failed to acquire lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:05.048525 1 leaderelection.go:354] lock is held by ceph-csi-rbd-provisioner-69cb97fb6-7sf5r_c5b8b501-4629-4d4f-acaa-3174459f35ee and has not yet expired I0320 10:25:05.048555 1 leaderelection.go:255] failed to acquire lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:08.206146 1 leaderelection.go:354] lock is held by ceph-csi-rbd-provisioner-69cb97fb6-7sf5r_c5b8b501-4629-4d4f-acaa-3174459f35ee and has not yet expired I0320 10:25:08.206167 1 leaderelection.go:255] failed to acquire lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:12.505545 1 leaderelection.go:260] successfully acquired lease kube-system/rbd.csi.ceph.com-kube-system [controller-runtime] log.SetLogger(...) was never called; logs will not be displayed. Detected at: > goroutine 96 [running]: > runtime/debug.Stack() > /usr/local/go/src/runtime/debug/stack.go:24 +0x5e > sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot() > /go/src/github.com/ceph/ceph-csi/vendor/sigs.k8s.io/controller-runtime/pkg/log/log.go:60 +0xcd > sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Enabled(0xc0006a0c40, 0x10?) > /go/src/github.com/ceph/ceph-csi/vendor/sigs.k8s.io/controller-runtime/pkg/log/deleg.go:111 +0x32 > github.com/go-logr/logr.Logger.Info({{0x27c4810?, 0xc0006a0c40?}, 0xc0005877a8?}, {0xc0006de3c0, 0x5b}, {0xc000918000, 0x6, 0x6}) > /go/src/github.com/ceph/ceph-csi/vendor/github.com/go-logr/logr/logr.go:276 +0x72 > sigs.k8s.io/controller-runtime/pkg/internal/recorder.(*Provider).getBroadcaster.func1.1(0xc00046af00) > /go/src/github.com/ceph/ceph-csi/vendor/sigs.k8s.io/controller-runtime/pkg/internal/recorder/recorder.go:104 +0x1e7 > k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher.func1() > /go/src/github.com/ceph/ceph-csi/vendor/k8s.io/client-go/tools/record/event.go:401 +0x11d > created by k8s.io/client-go/tools/record.(*eventBroadcasterImpl).StartEventWatcher in goroutine 87 > /go/src/github.com/ceph/ceph-csi/vendor/k8s.io/client-go/tools/record/event.go:387 +0x109 I0320 10:25:12.511509 1 reflector.go:289] Starting reflector *v1.PersistentVolume (9h9m18.353261575s) from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:105 I0320 10:25:12.511570 1 reflector.go:325] Listing and watching *v1.PersistentVolume from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:105 I0320 10:25:12.514780 1 reflector.go:351] Caches populated for *v1.PersistentVolume from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:105 I0320 10:25:12.654882 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:14.663620 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:16.698547 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:18.708013 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:20.715669 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:22.726674 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:24.736822 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:26.746649 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:28.755740 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:30.767344 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:32.785441 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:34.839030 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:36.932574 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:38.939961 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:40.951902 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:42.978052 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:44.988643 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:47.098921 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:49.187113 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:51.272895 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:53.285411 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:55.645270 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:57.707373 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:25:59.716334 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:01.732855 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:03.798490 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:05.809129 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:07.835245 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:11.237128 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:17.686766 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:19.854031 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:21.872842 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:23.886292 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:26.042042 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:28.154845 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:30.227750 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:32.351083 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:34.400862 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:36.549875 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:39.269400 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:41.278576 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:43.289851 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:45.301895 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:47.685415 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:48.114892 1 reflector.go:289] Starting reflector *v1.Secret (10h56m49.473323639s) from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:105 I0320 10:26:48.114923 1 reflector.go:325] Listing and watching *v1.Secret from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:105 I0320 10:26:48.136945 1 reflector.go:351] Caches populated for *v1.Secret from sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:105 I0320 10:26:48.238171 1 omap.go:89] got omap values: (pool="kube", namespace="", name="csi.volumes.default"): map[csi.volume.pvc-37c2b51a-54a4-4db2-b97d-570629bcccef:1a4370c8-3a79-445a-81a9-3ba8d3ac0930] I0320 10:26:48.239213 1 omap.go:89] got omap values: (pool="kube", namespace="", name="csi.volume.1a4370c8-3a79-445a-81a9-3ba8d3ac0930"): map[csi.imageid:37d7e558c272 csi.imagename:csi-vol-1a4370c8-3a79-445a-81a9-3ba8d3ac0930 csi.volname:pvc-37c2b51a-54a4-4db2-b97d-570629bcccef csi.volume.owner:default] I0320 10:26:48.318413 1 omap.go:89] got omap values: (pool="kube", namespace="", name="csi.volumes.default"): map[csi.volume.pvc-37c2b51a-54a4-4db2-b97d-570629bcccef:1a4370c8-3a79-445a-81a9-3ba8d3ac0930] I0320 10:26:48.319781 1 omap.go:89] got omap values: (pool="kube", namespace="", name="csi.volume.1a4370c8-3a79-445a-81a9-3ba8d3ac0930"): map[csi.imageid:37d7e558c272 csi.imagename:csi-vol-1a4370c8-3a79-445a-81a9-3ba8d3ac0930 csi.volname:pvc-37c2b51a-54a4-4db2-b97d-570629bcccef csi.volume.owner:default] I0320 10:26:49.696896 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:51.706089 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:54.315188 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:56.534423 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:26:58.676979 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:01.601297 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:03.850724 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:06.017947 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:10.257132 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:12.330606 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:15.090994 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:17.218829 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:19.423566 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:21.528787 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:25.574350 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:29.705811 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:32.736282 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:34.774710 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:36.897342 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:39.860294 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:42.434550 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:45.326598 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:47.623014 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:50.523953 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:53.947113 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:56.313573 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:27:58.891279 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system I0320 10:28:06.531098 1 leaderelection.go:281] successfully renewed lease kube-system/rbd.csi.ceph.com-kube-system E0320 10:28:18.532948 1 leaderelection.go:369] Failed to update lock: Put "https://10.96.0.1:443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rbd.csi.ceph.com-kube-system": context deadline exceeded I0320 10:28:18.533020 1 leaderelection.go:285] failed to renew lease kube-system/rbd.csi.ceph.com-kube-system: timed out waiting for the condition E0320 10:28:18.533132 1 controller.go:85] failed to start manager leader election lost E0320 10:28:18.533145 1 cephcsi.go:314] leader election lost