{"time": "2026-07-06T23:27:24.168673+00:00", "taskName": null, "msg": "Starting collector", "level": "INFO"}
{"time": "2026-07-06T23:27:24.168881+00:00", "taskName": null, "msg": "No folder annotation was provided, defaulting to k8s-sidecar-target-directory", "level": "WARNING"}
{"time": "2026-07-06T23:27:24.169200+00:00", "taskName": null, "msg": "Loading incluster config ...", "level": "INFO"}
{"time": "2026-07-06T23:27:24.169893+00:00", "taskName": null, "msg": "Config for cluster api at 'https://10.96.0.1:443' loaded...", "level": "INFO"}
{"time": "2026-07-06T23:27:24.170004+00:00", "taskName": null, "msg": "Unique filenames will not be enforced.", "level": "INFO"}
{"time": "2026-07-06T23:27:24.170228+00:00", "taskName": null, "msg": "5xx response content will not be enabled.", "level": "INFO"}
{"time": "2026-07-06T23:27:24.193725+00:00", "taskName": null, "msg": "Writing /etc/grafana/provisioning/datasources/datasource.yaml (ascii)", "level": "INFO"}
{"time": "2026-07-06T23:27:24.195804+00:00", "taskName": null, "msg": "Retrying (Retry(total=4, connect=9, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fbd96d76990>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/admin/provisioning/datasources/reload", "level": "WARNING"}
{"time": "2026-07-06T23:27:26.397259+00:00", "taskName": null, "msg": "Retrying (Retry(total=3, connect=8, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fbd98462390>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/admin/provisioning/datasources/reload", "level": "WARNING"}
{"time": "2026-07-06T23:27:30.798493+00:00", "taskName": null, "msg": "Retrying (Retry(total=2, connect=7, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fbd98462ba0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/admin/provisioning/datasources/reload", "level": "WARNING"}
{"time": "2026-07-06T23:27:39.600121+00:00", "taskName": null, "msg": "Retrying (Retry(total=1, connect=6, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fbd98463260>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/admin/provisioning/datasources/reload", "level": "WARNING"}
{"time": "2026-07-06T23:27:57.201782+00:00", "taskName": null, "msg": "Retrying (Retry(total=0, connect=5, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fbd984633b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/admin/provisioning/datasources/reload", "level": "WARNING"}
{"time": "2026-07-07T00:10:16.499725+00:00", "taskName": null, "msg": "ProtocolError when calling kubernetes: (\"Connection broken: InvalidChunkLength(got length b'', 0 bytes read)\", InvalidChunkLength(got length b'', 0 bytes read))\n", "level": "ERROR"}
{"time": "2026-07-07T00:10:16.500913+00:00", "taskName": null, "msg": "ProtocolError when calling kubernetes: (\"Connection broken: InvalidChunkLength(got length b'', 0 bytes read)\", InvalidChunkLength(got length b'', 0 bytes read))\n", "level": "ERROR"}
{"time": "2026-07-07T00:10:21.503563+00:00", "taskName": null, "msg": "Retrying (Retry(total=4, connect=9, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd99212330>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:10:21.503560+00:00", "taskName": null, "msg": "Retrying (Retry(total=4, connect=9, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd984cbe60>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:10:23.704405+00:00", "taskName": null, "msg": "Retrying (Retry(total=3, connect=8, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd992125d0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:10:23.704737+00:00", "taskName": null, "msg": "Retrying (Retry(total=3, connect=8, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd984cf0b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:10:28.106346+00:00", "taskName": null, "msg": "Retrying (Retry(total=2, connect=7, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd984cc0e0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:10:28.106400+00:00", "taskName": null, "msg": "Retrying (Retry(total=2, connect=7, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd99212840>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:10:36.907190+00:00", "taskName": null, "msg": "Retrying (Retry(total=1, connect=6, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd984cc350>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:10:36.907776+00:00", "taskName": null, "msg": "Retrying (Retry(total=1, connect=6, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd99212ab0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:10:54.509214+00:00", "taskName": null, "msg": "Retrying (Retry(total=0, connect=5, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd984cc5f0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:10:54.509693+00:00", "taskName": null, "msg": "Retrying (Retry(total=0, connect=5, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd99212b70>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:10:54.510544+00:00", "taskName": null, "msg": "MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd9b6f8bc0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n", "level": "ERROR"}
{"time": "2026-07-07T00:10:54.510612+00:00", "taskName": null, "msg": "MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd984cdbb0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n", "level": "ERROR"}
{"time": "2026-07-07T00:10:59.512727+00:00", "taskName": null, "msg": "Retrying (Retry(total=4, connect=9, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd984d7140>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:10:59.513027+00:00", "taskName": null, "msg": "Retrying (Retry(total=4, connect=9, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd984d08f0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:11:01.713411+00:00", "taskName": null, "msg": "Retrying (Retry(total=3, connect=8, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd99211910>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:11:01.713835+00:00", "taskName": null, "msg": "Retrying (Retry(total=3, connect=8, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd984d0950>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:11:06.114665+00:00", "taskName": null, "msg": "Retrying (Retry(total=2, connect=7, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd99210e00>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:11:06.115392+00:00", "taskName": null, "msg": "Retrying (Retry(total=2, connect=7, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd984d0c20>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:11:14.915925+00:00", "taskName": null, "msg": "Retrying (Retry(total=1, connect=6, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd99213140>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:11:14.916835+00:00", "taskName": null, "msg": "Retrying (Retry(total=1, connect=6, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd984d0ef0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:11:32.517511+00:00", "taskName": null, "msg": "Retrying (Retry(total=0, connect=5, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd992133b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:11:32.519175+00:00", "taskName": null, "msg": "MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd99213620>: Failed to establish a new connection: [Errno 111] Connection refused'))\n", "level": "ERROR"}
{"time": "2026-07-07T00:11:32.519172+00:00", "taskName": null, "msg": "Retrying (Retry(total=0, connect=5, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd984d11c0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:11:32.520917+00:00", "taskName": null, "msg": "MaxRetryError when calling kubernetes: HTTPSConnectionPool(host='10.96.0.1', port=443): Max retries exceeded with url: /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd984ce3c0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n", "level": "ERROR"}
{"time": "2026-07-07T00:11:37.521786+00:00", "taskName": null, "msg": "Retrying (Retry(total=4, connect=9, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd99218170>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:11:37.523627+00:00", "taskName": null, "msg": "Retrying (Retry(total=4, connect=9, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd984cb680>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:11:39.722528+00:00", "taskName": null, "msg": "Retrying (Retry(total=3, connect=8, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd992183b0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/configmaps?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:11:39.725711+00:00", "taskName": null, "msg": "Retrying (Retry(total=3, connect=8, read=5, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fbd984cb8f0>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/v1/namespaces/monitoring/secrets?labelSelector=grafana_datasource%3D1&timeoutSeconds=60&watch=True", "level": "WARNING"}
{"time": "2026-07-07T00:11:40.114043+00:00", "taskName": null, "msg": "ApiException when calling kubernetes: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': 'c67d97d0-0d66-4091-a0df-1e2ca1c4fda4', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '', 'X-Kubernetes-Pf-Prioritylevel-Uid': '', 'Date': 'Tue, 07 Jul 2026 00:11:40 GMT', 'Content-Length': '736'})\nHTTP response body: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"secrets is forbidden: User \\\\\"system:serviceaccount:monitoring:kube-prometheus-stack-grafana\\\\\" cannot watch resource \\\\\"secrets\\\\\" in API group \\\\\"\\\\\" in the namespace \\\\\"monitoring\\\\\": RBAC: [clusterrole.rbac.authorization.k8s.io \\\\\"system:discovery\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"system:service-account-issuer-discovery\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"system:public-info-viewer\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"kube-prometheus-stack-grafana-clusterrole\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"system:basic-user\\\\\" not found]\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"secrets\"},\"code\":403}\\n'\n\n", "level": "ERROR"}
{"time": "2026-07-07T00:11:40.114874+00:00", "taskName": null, "msg": "ApiException when calling kubernetes: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': '58d6c7af-e976-4ea4-aad7-c4539a9db59f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '', 'X-Kubernetes-Pf-Prioritylevel-Uid': '', 'Date': 'Tue, 07 Jul 2026 00:11:40 GMT', 'Content-Length': '745'})\nHTTP response body: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"configmaps is forbidden: User \\\\\"system:serviceaccount:monitoring:kube-prometheus-stack-grafana\\\\\" cannot watch resource \\\\\"configmaps\\\\\" in API group \\\\\"\\\\\" in the namespace \\\\\"monitoring\\\\\": RBAC: [clusterrole.rbac.authorization.k8s.io \\\\\"system:basic-user\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"system:discovery\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"system:service-account-issuer-discovery\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"system:public-info-viewer\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"kube-prometheus-stack-grafana-clusterrole\\\\\" not found]\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"configmaps\"},\"code\":403}\\n'\n\n", "level": "ERROR"}
{"time": "2026-07-07T00:11:40.129055+00:00", "taskName": null, "msg": "ApiException when calling kubernetes: (403)\nReason: Forbidden\nHTTP response headers: HTTPHeaderDict({'Audit-Id': '6528b3ba-24b8-4626-b269-725e83f8db45', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '', 'X-Kubernetes-Pf-Prioritylevel-Uid': '', 'Date': 'Tue, 07 Jul 2026 00:11:40 GMT', 'Content-Length': '736'})\nHTTP response body: b'{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"secrets is forbidden: User \\\\\"system:serviceaccount:monitoring:kube-prometheus-stack-grafana\\\\\" cannot watch resource \\\\\"secrets\\\\\" in API group \\\\\"\\\\\" in the namespace \\\\\"monitoring\\\\\": RBAC: [clusterrole.rbac.authorization.k8s.io \\\\\"system:service-account-issuer-discovery\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"system:public-info-viewer\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"kube-prometheus-stack-grafana-clusterrole\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"system:basic-user\\\\\" not found, clusterrole.rbac.authorization.k8s.io \\\\\"system:discovery\\\\\" not found]\",\"reason\":\"Forbidden\",\"details\":{\"kind\":\"secrets\"},\"code\":403}\\n'\n\n", "level": "ERROR"}